AR Matchmaking: The Compatibility of Musical Instruments with an AR Interface

Hyunkyung Shin, and Henrik von Coler

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

Augmented Reality (AR) interfaces offer new possibilities for musical expression by extending the capabilities of acoustic, electronic, and electroacoustic instruments. This study investigates the usability of the ARCube, an AR-based spatial audio controller, with twelve distinct musical instruments played by experienced musicians. We identify usability challenges specific to certain instruments, particularly for two-handed playing, as well as issues related to gesture recognition and cube stability. Our analysis shows that interaction patterns, such as cube placement, sound effect usage, and gesture strategies, vary significantly between instruments. These differences are driven by the physical form of the instruments, the required playing techniques, and user expectations for control and responsiveness. Based on these insights, we suggest directions for developing adaptable AR interfaces that better accommodate diverse instruments and support broader integration of AR technologies into musical practice.

Citation

Hyunkyung Shin, and Henrik von Coler. 2025. AR Matchmaking: The Compatibility of Musical Instruments with an AR Interface. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.15698952 [PDF]

BibTeX Entry

@article{nime2025_78,
 abstract = {Augmented Reality (AR) interfaces offer new possibilities for musical expression by extending the capabilities of acoustic, electronic, and electroacoustic instruments. This study investigates the usability of the ARCube, an AR-based spatial audio controller, with twelve distinct musical instruments played by experienced musicians. We identify usability challenges specific to certain instruments, particularly for two-handed playing, as well as issues related to gesture recognition and cube stability. Our analysis shows that interaction patterns, such as cube placement, sound effect usage, and gesture strategies, vary significantly between instruments. These differences are driven by the physical form of the instruments, the required playing techniques, and user expectations for control and responsiveness. Based on these insights, we suggest directions for developing adaptable AR interfaces that better accommodate diverse instruments and support broader integration of AR technologies into musical practice. },
 address = {Canberra, Australia},
 articleno = {78},
 author = {Hyunkyung Shin and Henrik von Coler},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.15698952},
 editor = {Doga Cavdir and Florent Berthaut},
 issn = {2220-4806},
 month = {June},
 numpages = {8},
 pages = {537--544},
 title = {AR Matchmaking: The Compatibility of Musical Instruments with an AR Interface},
 track = {Paper},
 url = {http://nime.org/proceedings/2025/nime2025_78.pdf},
 year = {2025}
}