In the third and final interview from the 2015 International Computer Music Conference (ICMC 2015), we had the privilege of speaking with Dr. Marco Buongiorno Nardelli. Dr. Nardelli is a distinguished research professor of physics and chemistry at the University of North Texas. He has been able to successfully merge his career as a computational materials physicist and his passion as a composer with his materialssoundmusic project. He is a member of the Initiative for Advance Research of Technology in the Arts, a fellow of the American Physical Society and the Institute of Physics, a founding member of the A Flow Consortium, and a Parma recordings artist.

Anderson: As a composer and a physicist, you have a passion for both music and physics. When did you first become interested in those fields?

I have many more years of professional life in physics than I have as a composer but the two things have always gone together.

Nardelli: I’ve been a musician way before I became a physicist actually because I started studying music as child maybe 6 or 7 years old. And, at a certain point I wanted to get into a professional career in music, and it was at the end of my high school. But, the situation with the conservatories in Italy was a little bit of a mess at the time. So, one thing and the other thing, and I decided to go in physics. I went into physics, but I always kept music at least on the side. I never stopped that.

Anderson: So would you consider yourself more of a physicist?

Nardelli: Right now, I would like to see myself as both. Clearly I have many more years of professional life in physics than I have as a composer but the two things have always gone together.

Anderson: So you have managed to combine these two interests, and I’d like to hear about that. Before we broach that subject, how about telling us a little bit about what you do in physics.

Nardelli: Yes, so as you said I’m a computational materials physicist which means that I study materials from a theoretical and computational point of view. We use computers to simulate the properties of materials on very microscopic levels. We are talking about atoms, electrons, and protons. Basically on that level.

[O]ne of our goals in the consortium is to develop databases of material properties that would allow scientists and engineers to find…materials that might be suitable for applications that have not been considered before.

In that way we are able to be predictive on properties of materials that might not have been synthesized yet or properties of materials under conditions that would be difficult to realize in a laboratory experiments.

In general we are able to get a lot of information that can be used to categorize the particular materials for a given application. When I talk about applications here, I mean any kind of electronics that you may carry in your hands or your purse or your bag. It’s made of advanced materials.

There is a need for materials that are able to push the technology farther, and so one of our goals in the consortium is to develop databases of material properties that would allow scientists and engineers to find – through the analysis of our data – materials that might be suitable for applications that have not been considered before.

So, right now, for instance, most of our technology is based on silicon. Silicon is material that was known for centuries or millennia. We’ve been using this for almost a century, and many of the applications that we would like to have or have already are not possible just with silicon. You need more advanced materials. You need to design more advanced materials to go beyond the technology that you have right now. So, that’s kind of the framework.

These databases that we develop using our theories and computational techniques are massive. Our database right now is almost a million entries of material entries. Actually you can go online on www.aflowlib.org and you have a periodic table and you click on the elements that you want in your material. If the material has been calculated then it pops up. It shows you the structure, the electronic properties, all the things that you need to know to decide if that composition is the right composition for your application.

So that’s kind of the work that we are doing, and this has been going on for a long time. We got a big push in 2011 with an initiative from the White House because the Obama Administration launched the Materials Genome Initiative. This is an initiative meant to accelerate material discovery and deployment from the research side to the application side in order to have this fast track into the commercial system as soon as possible. So, that’s where we fit in this picture.

Anderson: So, you have been able to take your background as a composer and a musician with this aflowlib repository of materials’ properties and marry them together in this project.

Nardelli: Yes. It’s something that I’ve been thinking about for quite some time. Actually, about a year ago I really started to work on this idea of taking this massive amount of data and using it to make music.

[A]bout a year ago I really started to work on this idea of taking this massive amount of data and using it to make music.

There are two aspects here. There is an aspect that is more scientific in a sense. That is the aspect of sonification. That is a way of realizing the data in a way that you can hear rather than see. The parallel of visualization for your eyes, you have sonification for your ears. One of the main uses of databases is you want to data mine the database because you want to look for particular elements.

One way of categorizing the elements is via visualizing some of the properties or, as I started to do, associate sounds with some of the properties so you can distinguish between two different materials not by just looking at them but by hearing the sound signature of the materials. The scientific drive here is to find ways of data analysis that are not just based on the visual but based on the aural.

The other aspect that is the more artist one is to use this flow of data as the building block or starting point for a composition process. The thinking is, or at least what I see is, you can always look at music as a data stream that is modified in real time. Even if you think about your scale — your scale is a data flow. You modify your scale and you have melodies which are modifications of the data flow. Now, instead of adding scales here or starting with a predefined set of elements, I take the data as they come out. I do some mapping to map the data onto pitches, for instance, or durations or intensities. Then I use that as my data stream that I manipulate to make a composition. So, that’s kind of the flow and so far I’m having a lot of fun doing it.

Anderson: So, you’re able to use data driven audio engineering both as a tool for sonification – for perceiving materials’ properties – as well as for composition.

You can hear the difference between data distributions that look very similar to each other. It would be very difficult to distinguish one from the other just by looking at them while you can hear an immediate difference. It’s very distinct.

Nardelli: Yes. I see there is this initial step that is more kind of the mapping. You have this data and you want to map them into something that can be heard, into sound. The mapping that I choose is very simple. I just map the data onto a media event that can be then read by software, basically.

I wrote all these interfaces that allow me to get the data, then translate them into sound, and drive them through a chain of applications, mainly MAX that is this code that all the electronic musicians use, and then into a workstation. In this process, I developed an interface that I call the data player that, on one hand, allows me to do the sonification of the data or at least some part of this data. On the other, it allows me to do the composition process.

There are examples on my website www.materialsoundmusic.com. One of the menu items is Materials. If you click on Materials you go to four examples of sonification of data for scientific purposes. You can hear the difference between data distributions that look very similar to each other. It would be very difficult to distinguish one from the other just by looking at them while you can hear an immediate difference. It’s very distinct.

Anderson: On that note, would you say there are advantages to using the auditory system to the visual system for some data sets?

Nardelli: That is actually something has been discussed extensively. Our hearing is much more sophisticated in a sense than our vision. We are able to distinguish many nuances by hearing that we couldn’t do just by looking at things.

We have no universal standards. We have not yet defined the equivalent of the Cartesian axis. We can plot things on a Cartesian axis, and we know what it means. We don’t really have something similar in sonification.

Paradoxically, I think the best would be if we could map data into smells because I think we have more receptors in our noses than in any other part of our bodies. Smells would be interesting, but I don’t know of any kinds of applications in this sense.

But, I think there is value. There are many examples of sonifications that have been done over the years. The typical ones are the ones that involve visually impaired people that need to guided by hearing other than vision.

There are examples of scientists that are blind, but they are to analyze data because the data has been sonified. I read about an astrophysicist. She was able to, using the sonification from data from telescopes or something, work with this in a scientific way.

There are not many examples because we are so much focused on visualization of data than sonification. The problem is that we need to train ourselves to use this for everyday scientific applications. We have no universal standards. We have not yet defined the equivalent of the Cartesian axis. We can plot things on a Cartesian axis, and we know what it means. We don’t really have something similar in sonification.

Anderson: With that in mind do you see sonification not only as a way of understanding or perceiving data but also as a predictive tool?

Nardelli: It would be nice. I don’t have an example, but if you train yourself to look for sound signatures that are descriptors of some property then that becomes a predictive tool because by hearing you can distinguish between systems, and you can say this is the system we need to use because it has right signature that the other ones don’t have.

Anderson: Do you think there is any reason why visual representations of data have dominated over audio ones?

We have a lot of data. There are data on anything and everything. I think visualizing things in just two dimensions is no longer enough. We need to find additional dimensions to represent the data so we can clearly navigate them.

Nardelli: One thing is that, so far, we are living in a time where data is becoming pervasive. We have a lot of data. There are data on anything and everything. I think visualizing things in just two dimensions is no longer enough. We need to find additional dimensions to represent the data so we can clearly navigate them.

There are examples of visualization caves, a kind of three dimensional environment where you visualize the data in three dimensions by walking inside an environment. But, I think that without adding other components, and I am convinced that sound is essential, we don’t go very far. Sound, has more dimensions than just a two dimensional piece of paper. So, you can really explore different regions of the data that you wouldn’t be able to do just by looking at a graph.

Anderson: As somebody who is just learning about sonification or data driven composition, do you have any recommended listening?

Nardelli: One of the problems of sonification is typically when you listen to a sonification of something it is not very interesting. Part of the problem is that most of the scientists are not composers and most of the composers are not scientists. So, it is very dry.

With what I’m trying to do, I have to admit I am manipulating the data. We are moving away from sonification here because I’m thinking of more from an artistic and aesthetic perspective. So, from the point of view of the scientific part, I think we can make the sound more interesting than just the dry sonification that we are used to.

We are very familiar with the universe and the stars, black holes. We have sonifications of astrophysical data. We have representations of climate data in music, or in sculpture, or in art; but, we don’t have anything for materials.

But I think that the richness of the material, and the way in which we can use it for artistic developments, has other applications and advantages that are maybe not completely scientific in the sense that they are not helping you to develop a model of the data. But they are compelling in the view that they are opening this huge space of research to people that are not scientists.

One of the things that I’m actively working on is to develop models of crystalline system materials that can be used as installations to make the audiences appreciate the fact that materials are important. We are very familiar with the universe and the stars, black holes. We have sonifications of astrophysical data. We have representations of climate data in music, or in sculpture, or in art; but, we don’t have anything for materials.

So, one of the motivations to start our project was: can we make something that we can then use as a tool to engage the public in the fact that anything we have and we use would not be there without the materials used to make it. I think music in this respect is very useful. One of the projects that I hope will work out right now if we can get the funding to do it is an installation where we have a gigantic crystal structure that is embedded into the sounds and the music that comes from the sonification and composition of the data that come from that material.

Anderson: Wow. I imagine you can cultivate a better intuition about the cells or atoms.

Nardelli: You are a physicist so you know what a material is, but most people don’t know what a material is. They might have studied chemistry in school. They know that there are atoms, but what is the material that we use to make all the circuitry of your iPhone? Or the glass of your computer? Or the cpu? Or anything that you have in your kitchen or your car? I think that is a huge market that has not been yet touched.

With what I’m trying to do, I have to admit I am manipulating the data. We are moving away from sonification here because I’m thinking of more from an artistic and aesthetic perspective. So, from the point of view of the scientific part, I think we can make the sound more interesting than just the dry sonification that we are used to.

Anderson: So, obviously, you background in music and composition has influenced how you want to engage the world with material science. Is the flip side true? When you put on your composer hat and you’re just thinking about writing music, has working with materials and all of this changed the way you view music more generally?

Nardelli: Yes, I think it did and it does in this more general sense of data driven composition. The fact that I start from a data flow. It does change the way that I approach the composition process. Now, the other advantage is knowing the data that I am using, because I’m an insider, I can manipulate them in ways that maybe other people wouldn’t think of. I know that if want to have a particular effect that maybe I should take data from here rather than from there. That is part of the freedom of the compositional process.

Anderson: Somebody kind of needs the background in both to first understand the data they want to sonify and then having the compositional background to turn that into a piece of music.

Nardelli: Yes, and this is part of the continuous effort interdisciplinary between the arts and the sciences that is being pushed on many different levels but isn’t really happening. I mean it’s happening, but on a very small scale. If you do a little research you will find visual artists that are using information and scientific research to incorporate in their art. There are many examples in biology for instance or climate or environmental sciences. But there is very little sound.

Anderson: Looking forward, where do see sound and science going into the next decade?

Nardelli: Well, one of the things that is essential and is something that is again part of the background on the work that we are doing on databases is quantity. So, one of my goals scientifically is to take our database, that is a database that is living in a sense – as we speak there are computers all over that are running calculations and that are added to the database – to make the sound signatures of all these data available in the database as part of the standard data that define the repository.

[S]onification as part of the datasets so that we will be able to run specific algorithmic approaches that would maybe be predictive…That is something that I think has to be done and has value;

So, that is one thing that I think is needed because when we talk about big data, the quality is in the quantity because when you have one entry you cannot do much with it. You cannot do any correlation and analysis in the big space. Here we have a big space. So, one of things that I want to do is keep pushing this aspect of sonification as part of the datasets so that we will be able to run specific algorithmic approaches that would maybe be predictive in a sense that using the data we can predict a new set of data. That is something that I think has to be done and has value; and, on the other hand, I will keep composing. That is more of the outreach aspect of this application.

Anderson: Dr. Nardelli it’s been a pleasure talking to you. Before we conclude I wanted to ask you is there anything that maybe we didn’t address that you’d like to talk about?

Nardelli: In the context of the International Computer Music Conference, there is a lot that can be applied to a project like mine. I think there is a lot of interest in this big data approach in music, not necessarily related to science, but anyways a big data approach. I think we are moving in the right direction. There are not many efforts but some efforts that I learned in research. It’s a good starting point.

Anderson: Dr. Nardelli thank you so much.