Please enjoy the 2nd in a series of interviews from the 2015 International Computer Music Conference (ICMC 2015) in Denton, Texas. Each year, this conference brings together scientists and artists from around the world to share their latest projects in the field of computer music.

In this episode, we interview Dr. Simon Lui, an assistant professor of the information systems technology and design pillar at Singapore University of Technology and Design. Lui received his PhD in computer science from the Hong Kong University of Science and Technology. He has extensive experience in mobile application development especially for audio applications. He started his own business in mobile application development in 2009. Dr. Lui has inventions on the iphone and ipad platforms, including number one best-selling apps. His work is widely reported by international media including CNN International and many other magazines, newspapers, and television programs. Dr. Lui is also a composer of computer music as well as an award-winning performer.

Tonks: Dr. Lui thank you so much for speaking with us today. Just to start off are a few basic questions. What is computer music and how did you first become interested in it?

I’m from the computer science background. I’m doing machine learning and artificial intelligence. So I’m using my skills to do something that I really like on musical sounds by using computer technology.

Lui: Okay, computer music is using a computer as a platform for music application. For example, for music performance or music analysis or music classification. So something that we cannot do as a human, for example, to produce some interesting sound or to deal with a large number of music we have to go to a computer and that’s why computer music. For me, I’m from the computer science background. I’m doing machine learning and artificial intelligence. So I’m using my skills to do something that I really like on music sounds by using computer technology.

Tonks: That’s very interesting. How does your work involve sonification?

Lui: Ah, sonification. My work mostly is using sound, using audio to help people and so that’s why I have to know what is sonification. What is the design, what is the principle of our audio signal and so that’s why I’m doing sound analysis and sound application and then use it for stroke patients and for deaf patients and for sports people in order to use music to enhance performance of some of their processes.

Tonks: That’s very interesting. Now I saw in one of the applications that you have involving replication of sound and your graded upon that. It’s very interesting to me because I’m learning Welsh and so approximating the intonation and approximating the way that you speak is very important in different languages. Do you see your work moving into the education of language?

[W]e are actually developing a new application using education so people can look… at the shape of all the different vowels so they can know what is the correct way of doing a British pronunciation ,or American accents, or even Cantonese or Mandarin.

Lui: Yeah ,that’s true. Because there are many kind of representations of sound for example audio representation and we also have vibration. When you touch the lips you’ll find that they are vibrating or you can look at the sound by doing some our conversions, for example, you can see the shape of the volume of your sound or the contour of your sound. So we are making use of some applications (iphone app) to visualize sounds so that you can look at it and then learn what is the correct shape with the shape of your sound. So we are actually developing a new application using education so people can look at ‘a’,’e’,’i’,’o’,’u’, at the shape of all the different vowels so they can know what is the correct way of doing a British pronunciation or American accents or even Cantonese or Mandarin. Yes, so we are actually making some small applications which can run efficiently to help people to look at the correct way of their pronunciation.

Tonks: That’s very interesting and I really think that’s groundbreaking and the future of that could be monumental for you and your colleagues. Tell us about how your work with biofeedback.

Lui: Yes. Because with music they have emotions. Usually with music they are expressive. So when people listen to music they have some response and we call that emotion and there are two kinds of emotion: induced emotion and perceived emotion. So how to classify them for example when you listen to a certain kind of music do you feel happy or do you feel sad. We have to evaluate it. So how to evaluate it? That’s what our project’s doing. We put some sensors, for example we put some EEG brain signal sensors here. I’ll put some skin contact sensors on your fingers and with the heart rate and respiration rate, we try to figure out all the biofeedback from your body when you listen to happy music and when you listen to sad music and compare the biofeedback to see how’s your body response tells us your emotion at that stage when you listen to the music. We are using those are signals to tell whether a person is happy or not or healthy or not or how their progress during the stroke rehabilitation process or the sports science status. Yeah, so we are doing our experiment in order to figure out what kind of music can give us the best results in doing such kind of music process.

Tonks: That’s just brilliant. You recently worked on a music therapy game designed for stroke rehabilitation. Can you tell us more about that.

[T]hey listen to the music and they have to do the exercise together with the music. In this case we find that certain kinds of music can help them to move faster, to move more efficiently, have better angles in their arm movements, and have better speed and better balance in the body when they’re walking.

Lui: Sure. Because there is a thing we call auditory motor synchronization which is something that’s inherent in people and that is present after they’re born. When they listen to some strong music they will synchronize and shake with the beat. It is something that we can not only picture. Everyone can do that. So for stroke patients since they have some problem in mobility they cannot move their arm or move their neck. So we wrote an application to help those patients to play them some music. So when we’re playing the music, they listen to the music and they have to do the exercise together with the music. In this case we find that are certain kinds of music can help them to move faster, to move more efficiently, have better angles in their arm movements, and have better speed and better balance in the body when they’re walking. Yeah, so that’s our project we have to choose the best kind of music to enhance their movement in the exercise.

Tonks: I find that very interesting how this application is also related to sport. I myself play football or soccer and a lot of times there’s a running soundtrack in my mind as the cadence. That’s very interesting.

So how is your work with sonification changed the way you think about music? In what way has your work changed your perception of music? Have you found which music that has touched you personally and been able to may be self-medicate, personal therapy, not only through your research but the application of your research to yourself?

We find out the results are the same but the participants tell us that they feel happier when using music itself instead of metronome. So for us, music research is not just the power, how effective or how accurate it is, but it’s also about… your experience when you’re using such a medium to help you do or achieve something.

Lui: I think it has a big impact on me for example in sport science we use something like a metronome. We can use a metronome to run faster. But at the same time we can also use the same piece of music of similar tempo to replay the music you follow the beat and you can also run faster. We find out the results are the same but the participants tell us that they feel happier when the music itself instead of metronome. So for us music research isn’t not just the power, how effective or how accurate it is, but it’s also about how you feel, how’s your response and how you experience when you’re using such a medium to help you do or achieve something. So for me I think music is something more than effectiveness more than accuracy. It is about how to help people to feel happier and how they are enjoying their life. It is about enjoyment, about cheers.

Tonks: That’s beautiful. Now do you see data-driven audio engineering more as a tool for music therapy or a tool for composition?

Lui: Oh, I think both because now it is the world of big data. People are using a set of data driven thing, something, they have a library, a database. They get some knowledge from the database and then they use the database for some other purposes. So actually, this kind of data driven model, a Markov model, a machine model – we can use it for composition because there’s some rules, something we can extract from it. We can use it for our music production but at the same time the tools can also have a lot of different kinds of applications. So the goal for us as professors and scholars we have to build a lot of tools. And then how to use the tools depends on users. So we have to make the work as scalable as possible so that they can transfer the knowledge to different fields of people so that the different people can use the same tools.

Tonks: So it has to be somewhat subjective but in the general sense it has to apply to the population.

Lui: Yeah, exactly.

Tonks: Do you think our auditory system has any kind of advantage over our visual system in interpreting or finding meaning in data?

[M]usic is something that I think is at a higher level – a higher semantic level. Other than what you can hear, you can sense the emotion, you can sense the layers, and you can sense the progression in some music. It gives you more information other than what you can see or what you can listen to in the audio spectrum.

Lui: Yeah, there are two ways. First is for disabled people for example for blind patients, actually that’s one of the papers I have for ICMC, for blind people they can identify direction by sound. So they turn left or turn right only by sound. And on the other hand for healthy persons sound can tell them additional information other than visual. For example something that is visual is 2D. To see 2D they will see the pixels or the kind of colors, whatever, but music it is something that i think it is at a higher level – a higher semantic level. Other than what you can hear, you can sense the emotion, you can sense the layers, and you can sense the progression in some music. It gives you more information other than what you can see or what you can listen to in the audio spectrum. So it is very interesting.

Tonks: So you would say that the auditory system is a multi-dimensional while the visual system is kind of two-dimensional?

Lui: I think so.

Tonks: Very interesting. Why do you think sound is a powerful therapeutic tool?

Lui: Because music induces emotion. When you listen to music the auditory motor synchronization will induce some reaction inside your brain in the alpha wave and also your brain and also your heart rate your overall content that’s inside your body. It is something that you cannot control. So it is something that can really help you to do something, that can enhance your motivation, to change your mind, change your attitude, and for some encouragement to push you to do something better, in terms of your performance in many kinds of processes and that’s the measure of music.

Tonks: That’s brilliant. Now I know that you are involved in composing as well as performing music. What kind of music do you enjoy listening to?

Lui: Oh, for me I like our wide range of music. But in particular I love a cappella. I love listening to classic choir as well as jazz music because this is music that I love to perform when I was in school I used to listen to many of these kinds and perform a lot of all. I have special interests in this kind of music — jazz, classical, and a cappella.

Tonks: Now with a cappella that’s very interesting to me because there aren’t many dimensions; it’s very succinct. There is no accompaniment. That’s very interesting. Could you expound on that?

[U]sing human voice we can do a bass, soprano, also do a guitar, many kind of strings, interesting sounds. And we can combine them together to create an interesting experience. I love a challenge. I love limitation. And love what I can achieve under this bound.

Lui: Yeah, because I myself love to work within some limitations, having to be challenged. For example, in a cappella you can only use a human vocal to achieve something which is very full, very complete. So you have to have some special skills in musical arrangement so that it can sound full. For example, for me I write a lot of a cappella music for my a cappella team in Hong Kong. We’ve done some competitions in Hong Kong. In one competition we became champions in Hong Kong. So that was fun because only using human voice we can do a bass, soprano, also do a guitar, many kind of strings, interesting sounds. And we can combine them together to create an interesting experience. So I love a challenge. I love limitation. And love what I can achieve under this bound.

Tonks: It’s very interesting that you enjoyed working with the restrictions of a very succinct form. A capella is something much different than other kinds of music. In addition to casual listening how does music and sound impact your daily life?

Lui: When I go to school, when I have to do revision I always have to listen to music because music helps me to concentrate and actually that is also imbues some of my work. There are some moves in music that can help people to concentrate. So music is very important in my life. It gives me motivation, for leisure, and it’s the best accompaniment when I’m bored and when I need something more to enhance, to motivate me. Music is very important in my life actually.

Tonks: That’s very interesting. Now where do you see your work with sound and music therapy going into the next decade?

Lui: Now people have very limited knowledge in this field. Most of the recent work that is used in some kind of therapy but our most of the work has lack of justification. That is people know that this kind of process can help you to do music therapy, but they don’t know why. For example, what kind of music elements can help them, what kind of music, songs, what kind of tempo, whatever can help a music therapy process to become more effective. So I think in the next decade people have to find a justification, find the reasons, by some carefully designed listening test, clinical tests in order to justify. After that we can use the knowledge to help more people in order to use music in music therapy.

Tonks: So music has been defined in many different ways by many different people. How would you define music?

[T]he boundaries between all these 3 classes — music, sound, and noise are becoming less clear. Because for contemporary music people tend to use sound and even random noise in order to compose something as music.

Lui: Music, sound, and noise they have a very clear separation of classification. For music it is something they have a certain format, for example a melody, a baseline, a chord progression or whatever; and then on the other hand, we have something we call sound. That is any kind of audio thing that we can listen by ear but some of them are more structured; on the other hand, we have something we call noise which is totally random like white noise, purple noise. I can say for now the boundaries between all these 3 classes — music, sound, and noise are becoming less clear. Because for contemporary music people tend to use sound and even random noise in order to compose something as music. The reason is they find that this kind of audio composition is interesting and they can infuse some emotion and there are some different kind of interesting response. People are trying to explore into a new field. So I would say these three categories, they’re separated but now they are trying to bound together.

Tonks: So you would agree that one of the definitions of music is everything that one listens to with the intention of listening to music? So with with that definition it seems as though those boundaries have been nullified. It’s been a very interesting talking to you. Before we conclude the interview I want to give you a chance to direct the conversation. So, on that note, is there anything that we didn’t discuss now that you’d like to address?

Lui: Before we have more computer applications, people find that music is something very difficult, very high level, and only a musician can do composition. Only a rich people can buy music or go to a musical concert, but for me I think that’s not true. I think music should be for everyone even for laymen, for my mother. I think we should write some applications such as my mother even can enjoy writing music by using some tools, something that can help them to understand the basics even if they don’t understand the music principles, but they can use tools to write some simple music that they can enjoy. So I think that’s my job — to do something.

Tonks: So you see music compelling as an educational tool?

Lui: Yeah, an education tool and for self-enjoyment and which more people are going to enjoy with less barriers.

Tonks: Very interesting. We really appreciate your time and thank you so much for what you do and I wish you all the best in your future endeavors.