Feel What You Don't Hear: A New Framework for Non-aural Music Experiences

Aoi Uyama, Danny Hynds, Dingding Zheng, George Chernyshov, Tatsuya Saito, Kai Kunze, and Kouta Minamizawa

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract:

Just as the way a performer is moved differs even among audiences who have the same impression of the performance, the sensations and experiences felt by the performers themselves and the audiences' experiences also differ. The purpose of this research is to create a new listening experience by analyzing and extracting the performer's introspection of rests, groove, and rhythm, and physically presenting it to the audience. Although these elements are important in shaping music, they are not always directly expressed as auditory sounds. Our hypothesis is that this introspection, such as a sense of rhythm and groove, is latent and observable in physiological states such as breathing and heartbeat. By sensing and presenting them to the audience, music appreciation that includes introspection could become possible. In other words, by sensing and presenting introspection to the audience, the music listening experience itself can be redesigned to include a physicality that is closer to the performer's experience of the music, rather than being passive in an auditory sense. In this study, preliminary experiments were conducted on the extraction of the performer's introspection, and a device was designed to present it to the audience.

Citation:

Aoi Uyama, Danny Hynds, Dingding Zheng, George Chernyshov, Tatsuya Saito, Kai Kunze, and Kouta Minamizawa. 2023. Feel What You Don't Hear: A New Framework for Non-aural Music Experiences. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.11189288

BibTeX Entry:

  @inproceedings{nime2023_79,
 abstract = {Just as the way a performer is moved differs even among audiences who have the same impression of the performance, the sensations and experiences felt by the performers themselves and the audiences' experiences also differ. The purpose of this research is to create a new listening experience by analyzing and extracting the performer's introspection of rests, groove, and rhythm, and physically presenting it to the audience. Although these elements are important in shaping music, they are not always directly expressed as auditory sounds.
Our hypothesis is that this introspection, such as a sense of rhythm and groove, is latent and observable in physiological states such as breathing and heartbeat. By sensing and presenting them to the audience, music appreciation that includes introspection could become possible. In other words, by sensing and presenting introspection to the audience, the music listening experience itself can be redesigned to include a physicality that is closer to the performer's experience of the music, rather than being passive in an auditory sense. In this study, preliminary experiments were conducted on the extraction of the performer's introspection, and a device was designed to present it to the audience.},
 address = {Mexico City, Mexico},
 articleno = {79},
 author = {Aoi Uyama and Danny Hynds and Dingding Zheng and George Chernyshov and Tatsuya Saito and Kai Kunze and Kouta Minamizawa},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.11189288},
 editor = {Miguel Ortiz and Adnan Marquez-Borbon},
 issn = {2220-4806},
 month = {May},
 numpages = {6},
 pages = {560--565},
 title = {Feel What You Don't Hear: A New Framework for Non-aural Music Experiences},
 track = {Work in Progress},
 url = {http://nime.org/proceedings/2023/nime2023_79.pdf},
 year = {2023}
}