When listening to Michael Jackson, Elvis Presley or Raphael, how long did it take us to associate their music with the movements that made them eternal on stage? When listening to Camarón, can we imagine them without closing our eyes and making a moaning face? A guitar solo by Jimi Hendrix, does it make sense to imagine him sitting with the guitar on his knees? And it is that music is much more than a good letter and melody, it is an interaction between everything we feel.
Tejaswinee Kelkar is a singer and researcher at the Department of Musicology at the University of Oslo who presented her thesis in 2019, under the suggestive title Computational analysis of melodic contour and body movement . As Kelkar herself says, “think about how you feel when someone sings really high notes […], what we notice is the effort that the singer is making”. We are able to recognize the pattern because it is a physiological issue, it is not necessary to see the singer. The researcher has studied the gestures that people make when listening to music. These gestures include facial expressions and leg position, but there are more elements that affect the listening experience. Somehow we keep an authentic library of gestures that give a unique value to the music of each performer. If you are of a certain age, think of Lola Flores and try to imagine her singing sitting on a chair. It is impossible.
Tejaswinee Kelkar herself is a singer and the origin of her research is in her own experience. He noticed how we use our hands differently when singing western or Indian music. When she was little she learned to sing North Indian music. There they used hand gestures for the children to learn to sing and, when they went up on stage, they continued to do so. But when he studied classical western singing and waved his hands, he was told that this was not how it was done. Kelkar believed that the movements of the hands gave him some kind of anatomical assistance to be able to use his voice and, therefore, some gestures were suitable for one type of music and not for another.
Music plays in all of our senses because it is multimodal, says Kelkar. Both space and time are very important in music. In order to better understand how music is perceived spatially, Kelkar has conducted several experiments. In one of these experiments, he asked participants to draw or explain what they were visualizing in their minds while repeatedly listening to a piece of music. “Several participants described or drew the music as a wave that passed by,” explains Kelkar. Other participants, if they noticed the melody repeating itself, drew circles.
In another experiment, participants were asked to move their arms according to what they heard. The methodology followed by the researcher was to record each participation and complement it with motion capture technology to detect patterns.
The relationship between what we see and what we hear is not new. Language researchers have been studying this connection for a long time and it is common in nursery schools. “It has been observed as a linguistic phenomenon, including the so-called McGurk effect,” explains Kelkar. This effect was first described by McGurk and MacDonald in a 1976 Nature article with a title that says it all: Listen to lips and read voices. The effect is about how we can hear a sound while looking at a face expressing a different sound. In this case we hear a third sound, which can be intermediate. For example, if the syllables “ba-ba” are emitted over the movements of the lips of “ga-ga”, the perception is of “da-da”. Kelkar wondered if the same thing happens when we sing, so his thesis was extended to an investigation with his colleagues Bruno Laeng and Sarjo Kuyateh.
“In fact, we found signs that what a singer does with his face affects the way people perceive a melody. The interval between two tones can sound different if the singer’s gestures vary ”.
With the data obtained and artificial intelligence, one can think of a future technology that recognizes music through the singer’s movements:
“Imagine a technology similar to the Shazam music recognition App, but scanning movements instead of sounds. If you made gestures coinciding with Happy Birthday, this App would search the song for you.”
Kelkar’s idea is not entirely new, as it has a certain similarity to the Parsons code, which is a method of identifying music based on melodic contours. It was Denys Parsons who paid attention to this aspect in 1975, with a book entitled Directory of melodies and musical themes . He analyzed no less than 15,000 pieces in his work and grouped them by melodic contours. A melodic outline is a structural layout of the heights and positions of the melody of a musical work. To do this, the most relevant notes of a melody are taken into account: the initial and final note of a phrase, the lowest and highest note of a phrase, the notes that are in strong beats of a measure, the notes accentuated with dynamics. , pronounced syncopations or anticipated notes and the longest notes compared to the surrounding notes.
The researcher herself has reported on her twitter account that the data and videos are available online with free access.
