Audionce: An Audiovisual Improvisation Environment Co-Designed by Deaf and Hearing Designers

Alon Ilsar, Matt Hughes, Yuka Maruyama, and Andrew Johnston

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

Audionce is an audiovisual installation designed to support Deaf and hearing musicians and non-musicians to experience collaborative musical improvisation through multimodal interaction.The installation challenges the assumption that improvisation is primarily an auditory practice by reframing musical agency through gesture, vibration, touch, and visual feedback. The system combines a physical array of loudspeakers and a subwoofer with a virtual `mirror' environment constructed from point-cloud scans of the same speakers.Using depth-camera tracking, animated lines connect participants to the speakers as their presence creates audiovisual content.As participants move through the space, additional audiovisual layers emerge, including drones, spatially responsive ostinati, and tactile kick-drum manipulation activated through direct contact with speaker surfaces. This paper presents the design motivations, system architecture, interaction mappings, artistic reflections.Audionce proposes a model for inclusive musical interaction that treats improvisation as a shared, embodied, and multi-sensory experience rather than one constrained by hearing ability.

Citation

Alon Ilsar, Matt Hughes, Yuka Maruyama, and Andrew Johnston. 2026. Audionce: An Audiovisual Improvisation Environment Co-Designed by Deaf and Hearing Designers. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.20784064 [PDF]

BibTeX Entry

@inproceedings{nime2026_13,
 abstract = {Audionce is an audiovisual installation designed to support Deaf and hearing musicians and non-musicians to experience collaborative musical improvisation through multimodal interaction.The installation challenges the assumption that improvisation is primarily an auditory practice by reframing musical agency through gesture, vibration, touch, and visual feedback. The system combines a physical array of loudspeakers and a subwoofer with a virtual `mirror' environment constructed from point-cloud scans of the same speakers.Using depth-camera tracking, animated lines connect participants to the speakers as their presence creates audiovisual content.As participants move through the space, additional audiovisual layers emerge, including drones, spatially responsive ostinati, and tactile kick-drum manipulation activated through direct contact with speaker surfaces. This paper presents the design motivations, system architecture, interaction mappings, artistic reflections.Audionce proposes a model for inclusive musical interaction that treats improvisation as a shared, embodied, and multi-sensory experience rather than one constrained by hearing ability.},
 address = {London, United Kingdom},
 articleno = {13},
 author = {Alon Ilsar and Matt Hughes and Yuka Maruyama and Andrew Johnston},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.20784064},
 editor = {Benedict Gaster and João Tragtenberg and Anna Xambó and Tom Mitchell},
 issn = {2220-4806},
 month = {June},
 note = {},
 numpages = {8},
 pages = {101--108},
 presentation-video = {https://youtu.be/JAaPPyDLVsY},
 title = {Audionce: An Audiovisual Improvisation Environment Co-Designed by Deaf and Hearing Designers},
 track = {Paper},
 url = {http://nime.org/proceedings/2026/nime2026_13.pdf},
 year = {2026}
}