Probing Latent Space Interactions with Real-time Generative Audio Models Through a Physical Controller

Domenico Stefani, Francesco Dal Rì, and Luca Turchet

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

Recent advances have made deep generative models practical for live music performance. Moreover, latent spaces are increasingly exposed as sites of musical interaction. However, their use as control substrates remains overall under-explored as a first-class design problem. We propose a tangible, reconfigurable controller, conceived as a design probe for investigating the tradeoff between exploration and reproducibility affordances across different mapping and visual feedback strategies. The system is demonstrated through three use cases for as many representative models for real-time synthesis of audio or symbolic music: RAVE, for neural synthesis with high-dimensional opaque latents; MT-GEN_DDSP, exploiting a known latent space with labeled timbre cluster; and GrooveTransformer, for semantically-structured rhythm pattern generation. Each use-case employs distinct sensor-to-latent mappings and visual feedback approaches tailored to the model’s characteristics, ranging from 3D latent traversal with interaction heatmaps, to semantic dimension mapping with timbre cluster visualization, to descriptor-guided space navigation. By treating latent spaces as explicit interaction surfaces, rather than implementation details, this work contributes to ongoing discussions about controllability, legibility, and appropriation in machine learning-based musical instruments.

Citation

Domenico Stefani, Francesco Dal Rì, and Luca Turchet. 2026. Probing Latent Space Interactions with Real-time Generative Audio Models Through a Physical Controller. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.20784186 [PDF]

BibTeX Entry

@inproceedings{nime2026_58,
 abstract = {Recent advances have made deep generative models practical for live music performance. Moreover, latent spaces are increasingly exposed as sites of musical interaction. However, their use as control substrates remains overall under-explored as a first-class design problem. We propose a tangible, reconfigurable controller, conceived as a design probe for investigating the tradeoff between exploration and reproducibility affordances across different mapping and visual feedback strategies. The system is demonstrated through three use cases for as many representative models for real-time synthesis of audio or symbolic music: RAVE, for neural synthesis with high-dimensional opaque latents; MT-GEN_DDSP, exploiting a known latent space with labeled timbre cluster; and GrooveTransformer, for semantically-structured rhythm pattern generation. Each use-case employs distinct sensor-to-latent mappings and visual feedback approaches tailored to the model’s characteristics, ranging from 3D latent traversal with interaction heatmaps, to semantic dimension mapping with timbre cluster visualization, to descriptor-guided space navigation. By treating latent spaces as explicit interaction surfaces, rather than implementation details, this work contributes to ongoing discussions about controllability, legibility, and appropriation in machine learning-based musical instruments.},
 address = {London, United Kingdom},
 articleno = {58},
 author = {Domenico Stefani and Francesco Dal Rì and Luca Turchet},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.20784186},
 editor = {Benedict Gaster and João Tragtenberg and Anna Xambó and Tom Mitchell},
 issn = {2220-4806},
 month = {June},
 note = {},
 numpages = {8},
 pages = {492--499},
 title = {Probing Latent Space Interactions with Real-time Generative Audio Models Through a Physical Controller},
 track = {Paper},
 url = {http://nime.org/proceedings/2026/nime2026_58.pdf},
 year = {2026}
}