Extended Reality Audio-Visual Instruments: Design Framework and Case Study

Esther Gruy, and Florent Berthaut

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

Scientists, artists and performers have explored the relationship between visuals and sound on numerous occasions, to try and find/define ways to fuse both sensory experiences together. Immersive technologies further amplify the possibilities for multimodal expression in performance contexts. While previous frameworks exist for the design of immersive musical instruments and audio-visual instruments independently, we believe that their combination raises specific concerns, which might prevent the exploration of expressive opportunities.In this article, we define the concept of Extended Reality Audio-Visual Instruments (XRAVI) and propose a framework for their analysis and design, based on the literature on audio-visual and extended reality instruments. We evaluate it through a collaborative autoethnographic work, where an initial shared instrument has been developed and practised following two different approaches: audio-first and visual-first. From it, we derive insights and guidelines for the design of XRAVI.

Citation

Esther Gruy, and Florent Berthaut. 2026. Extended Reality Audio-Visual Instruments: Design Framework and Case Study. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.20784281 [PDF]

BibTeX Entry

@inproceedings{nime2026_93,
 abstract = {Scientists, artists and performers have explored the relationship between visuals and sound on numerous occasions, to try and find/define ways to fuse both sensory experiences together. Immersive technologies further amplify the possibilities for multimodal expression in performance contexts. While previous frameworks exist for the design of immersive musical instruments and audio-visual instruments independently, we believe that their combination raises specific concerns, which might prevent the exploration of expressive opportunities.In this article, we define the concept of Extended Reality Audio-Visual Instruments (XRAVI) and propose a framework for their analysis and design, based on the literature on audio-visual and extended reality instruments. We evaluate it through a collaborative autoethnographic work, where an initial shared instrument has been developed and practised following two different approaches: audio-first and visual-first. From it, we derive insights and guidelines for the design of XRAVI.},
 address = {London, United Kingdom},
 articleno = {93},
 author = {Esther Gruy and Florent Berthaut},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.20784281},
 editor = {Benedict Gaster and João Tragtenberg and Anna Xambó and Tom Mitchell},
 issn = {2220-4806},
 month = {June},
 note = {},
 numpages = {10},
 pages = {788--797},
 presentation-video = {https://youtu.be/o85jwAKOD-w},
 title = {Extended Reality Audio-Visual Instruments: Design Framework and Case Study},
 track = {paper},
 url = {http://nime.org/proceedings/2026/nime2026_93.pdf},
 year = {2026}
}