From Improvised Movement to Musical Improvisation - Using Machine-Learning to Create Personalized Instruments for Dancers

Daniel Bisig, Alexander Oknupik, Johannes Schneider, Diane Gemsch, Eleni Mylona, and Tim Winkler

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

The paper presents the development and preliminary evaluation of a machine-learning-based digital instrument that translates a dancer’s bodily movements into music. It is trained on motion-capture and audio recordings of professional dancers improvising solo to music, thereby learning cross-modal correspondences between movement and sound. In performance, a dancer can then use the instrument as a highly personalized tool for generating music through bodily gestures. A transdisciplinary team of machine-learning researchers and dancers leads the development and evaluation, following a practice-led approach aligned with the dancers’ artistic interests. This includes selecting the movement and musical material for training and testing, assessing the instrument’s creative usability, and integrating it into rehearsals and the creation of new performance works.

Citation

Daniel Bisig, Alexander Oknupik, Johannes Schneider, Diane Gemsch, Eleni Mylona, and Tim Winkler. 2026. From Improvised Movement to Musical Improvisation - Using Machine-Learning to Create Personalized Instruments for Dancers. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.20784415 [PDF]

BibTeX Entry

@inproceedings{nime2026_135,
 abstract = {The paper presents the development and preliminary evaluation of a machine-learning-based digital instrument that translates a dancer’s bodily movements into music. It is trained on motion-capture and audio recordings of professional dancers improvising solo to music, thereby learning cross-modal correspondences between movement and sound. In performance, a dancer can then use the instrument as a highly personalized tool for generating music through bodily gestures. A transdisciplinary team of machine-learning researchers and dancers leads the development and evaluation, following a practice-led approach aligned with the dancers’ artistic interests. This includes selecting the movement and musical material for training and testing, assessing the instrument’s creative usability, and integrating it into rehearsals and the creation of new performance works.},
 address = {London, United Kingdom},
 articleno = {135},
 author = {Daniel Bisig and Alexander Oknupik and Johannes Schneider and Diane Gemsch and Eleni Mylona and Tim Winkler},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.20784415},
 editor = {Benedict Gaster and João Tragtenberg and Anna Xambó and Tom Mitchell},
 issn = {2220-4806},
 month = {June},
 note = {},
 numpages = {9},
 pages = {1108--1116},
 title = {From Improvised Movement to Musical Improvisation - Using Machine-Learning to Create Personalized Instruments for Dancers},
 track = {paper},
 url = {http://nime.org/proceedings/2026/nime2026_135.pdf},
 year = {2026}
}