Performing with the inclusive machine: An interdisciplinary roadmap for the design of AI collaborative musical instruments

Pablo Mollenhauer, and Alejandra Pérez Núñez

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

Computer musical instruments that use traditional programming paradigms have leaned towards predictability and direct causality based on deterministic mappings, rule-based synthesis, and explicit parameter control. Many studies have implemented instruments using machine learning for mapping and sound synthesis, but yet, few studies have explored the implications of these technologies on agency, causality, and power relations in the design process and performance practice. This paper explores the political-agential balances and aesthetics that emerge when machine learning technologies are integrated into the design process of computer music instruments. The hypothesis is that control and agency is distributed by machine learning's algorithms, in which designer and performer, have to navigate through the feedback system made of gestures, latent space and spatialized sound. The methodology comprises several chained machine learning stages: a single pose tracking system driven by pre-trained models, a supervised neural network for mapping parameters of the latent space of models for sound synthesis. The design stage is informed by first-person accounts of experience using micro-phenomenology.

Citation

Pablo Mollenhauer, and Alejandra Pérez Núñez. 2026. Performing with the inclusive machine: An interdisciplinary roadmap for the design of AI collaborative musical instruments. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.20784202 [PDF]

BibTeX Entry

@inproceedings{nime2026_64,
 abstract = {Computer musical instruments that use traditional programming paradigms have leaned towards predictability and direct causality based on deterministic mappings, rule-based synthesis, and explicit parameter control. Many studies have implemented instruments using machine learning for mapping and sound synthesis, but yet, few studies have explored the implications of these technologies on agency, causality, and power relations in the design process and performance practice. This paper explores the political-agential balances and aesthetics that emerge when machine learning technologies are integrated into the design process of computer music instruments. The hypothesis is that control and agency is distributed by machine learning's algorithms, in which designer and performer, have to navigate through the feedback system made of gestures, latent space and spatialized sound. The methodology comprises several chained machine learning stages: a single pose tracking system driven by pre-trained models, a supervised neural network for mapping parameters of the latent space of models for sound synthesis. The design stage is informed by first-person accounts of experience using micro-phenomenology.},
 address = {London, United Kingdom},
 articleno = {64},
 author = {Pablo Mollenhauer and Alejandra Pérez Núñez},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.20784202},
 editor = {Benedict Gaster and João Tragtenberg and Anna Xambó and Tom Mitchell},
 issn = {2220-4806},
 month = {June},
 note = {},
 numpages = {7},
 pages = {542--548},
 presentation-video = {https://youtu.be/Sc9RZbz9JkA},
 title = {Performing with the inclusive machine: An interdisciplinary roadmap for the design of AI collaborative musical instruments},
 track = {Paper},
 url = {http://nime.org/proceedings/2026/nime2026_64.pdf},
 year = {2026}
}