A Web Interface for Real-Time Interaction with Machine Learning in Musical Performance

Hongzhe Zhu, and Charles Martin

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

Interactive machine learning systems are increasingly enabling new forms of real-time musical collaboration between human performers and artificial intelligence, transforming how musicians access and manipulate performance data. However, many existing tools function as black boxes, obscuring the internal logic of the generative models and limiting the performer's sense of agency and trust. This research addresses this challenge by developing a transparent, web-based interface for an interactive machine learning system which visualises the state of mixture density recurrent neural networks (MDRNNs) in real-time. We present the design and evaluation of this system, detailing how its architecture and two-way visualisation strategy support multiple interaction paradigms. An exploratory user study suggests that while the conceptual complexity of model training presents a learning curve for novices, the system's real-time visual feedback can support user trust and enable expressive co-creation. We conclude that exposing the data provenance and generative processes of AI systems can help transform them from opaque tools into intelligible musical partners. This work contributes a framework for human-centered AI interface design in the NIME community, suggesting that transparency is an important design factor for creative agency in intelligent instruments.

Citation

Hongzhe Zhu, and Charles Martin. 2026. A Web Interface for Real-Time Interaction with Machine Learning in Musical Performance. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.20784312 [PDF]

BibTeX Entry

@inproceedings{nime2026_102,
 abstract = {Interactive machine learning systems are increasingly enabling new forms of real-time musical collaboration between human performers and artificial intelligence, transforming how musicians access and manipulate performance data. However, many existing tools function as black boxes, obscuring the internal logic of the generative models and limiting the performer's sense of agency and trust. This research addresses this challenge by developing a transparent, web-based interface for an interactive machine learning system which visualises the state of mixture density recurrent neural networks (MDRNNs) in real-time. We present the design and evaluation of this system, detailing how its architecture and two-way visualisation strategy support multiple interaction paradigms. An exploratory user study suggests that while the conceptual complexity of model training presents a learning curve for novices, the system's real-time visual feedback can support user trust and enable expressive co-creation. We conclude that exposing the data provenance and generative processes of AI systems can help transform them from opaque tools into intelligible musical partners. This work contributes a framework for human-centered AI interface design in the NIME community, suggesting that transparency is an important design factor for creative agency in intelligent instruments.},
 address = {London, United Kingdom},
 articleno = {102},
 author = {Hongzhe Zhu and Charles Martin},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.20784312},
 editor = {Benedict Gaster and João Tragtenberg and Anna Xambó and Tom Mitchell},
 issn = {2220-4806},
 month = {June},
 note = {},
 numpages = {8},
 pages = {860--867},
 presentation-video = {https://youtu.be/roAjjsT5uO0},
 title = {A Web Interface for Real-Time Interaction with  Machine Learning in Musical Performance},
 track = {paper},
 url = {http://nime.org/proceedings/2026/nime2026_102.pdf},
 year = {2026}
}