Emotion and Expressivity in Music Performance: A Multimodal Approach

Natalia Kotsani, Spyros Kantarelis, Vassilis Lyberatos, Edmund Dervakos, and Giorgos Stamou

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

The relationship between musical performance, perceived, intended, and induced emotion by audience and musician is a complex topic that has been studied for centuries. In the era of Artificial Intelligence (AI), this topic becomes particularly important for the development of applications that assist in the study and creative aspects of musical performance, enhancing both the interpretative process and emotional expression. In this paper we present a protocol for creating datasets that contain live performances of music compositions, continuous and categorical emotion annotations from the audience and the performing musician, in addition to an array of biosignals recorded from performers and listeners. The protocol is designed to contain a variety of musical contexts, from rigid etudes simulating practice to free improvisation emulating pure expression. It is replicable and agnostic of instrumentation, and its main purpose is to facilitate the development of AI applications that will enhance musical expression. We provide the design and details of the protocol, preliminary results from its implementation with professional musicians and we discuss limitations and potential future research directions.

Citation

Natalia Kotsani, Spyros Kantarelis, Vassilis Lyberatos, Edmund Dervakos, and Giorgos Stamou. 2026. Emotion and Expressivity in Music Performance: A Multimodal Approach. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.20784368 [PDF]

BibTeX Entry

@inproceedings{nime2026_118,
 abstract = {The relationship between musical performance, perceived, intended, and induced emotion by audience and musician is a complex topic that has been studied for centuries. In the era of Artificial Intelligence (AI), this topic becomes particularly important for the development of  applications that assist in the study and creative aspects of musical performance, enhancing both the interpretative process and emotional expression. In this paper we present a protocol for creating datasets that contain live performances of music compositions, continuous and categorical emotion annotations from the audience and the performing musician, in addition to an array of biosignals recorded from performers and listeners. The protocol is designed to contain a variety of musical contexts, from rigid etudes simulating practice to free improvisation emulating pure expression. It is replicable and agnostic of instrumentation, and its main purpose is to facilitate the development of AI applications that will enhance musical expression. We provide the design and details of the protocol, preliminary results from its implementation with professional musicians and we discuss limitations and potential future research directions.},
 address = {London, United Kingdom},
 articleno = {118},
 author = {Natalia Kotsani and Spyros Kantarelis and Vassilis Lyberatos and Edmund Dervakos and Giorgos Stamou},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.20784368},
 editor = {Benedict Gaster and João Tragtenberg and Anna Xambó and Tom Mitchell},
 issn = {2220-4806},
 month = {June},
 note = {},
 numpages = {7},
 pages = {972--978},
 title = {Emotion and Expressivity in Music Performance: A Multimodal Approach},
 track = {paper},
 url = {http://nime.org/proceedings/2026/nime2026_118.pdf},
 year = {2026}
}