Cubing Sound: Designing a NIME for Head-mounted Augmented Reality

Yichen Wang, and Charles Martin

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract:

We present an empirical study of designing a NIME for the head-mounted augmented reality (HMAR) environment. In the NIME community, various sonic applications have incorporated augmented reality (AR) for sonic experience and audio production. With this novel digital form, new opportunities for musical expression and interface are presented. Yet few works consider whether and how the design of the NIME will be affected given the technology’s affordance. In this paper, we take an autobiographical design approach to design a NIME in HMAR, exploring what is a genuine application of AR in a NIMEs and how AR mediates between the performer and sound as a creative expression. Three interface prototypes are created for a frequency modulation synthesis system. We report on their design process and our learning and experiences through self-usage and improvisation. Our designs explore free-hand and embodied interaction in our interfaces, and we reflect on how these unique qualities of HMAR contribute to an expressive medium for sonic creation.

Citation:

Yichen Wang, and Charles Martin. 2022. Cubing Sound: Designing a NIME for Head-mounted Augmented Reality. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.21428/92fbeb44.b540aa59

BibTeX Entry:

  @inproceedings{NIME22_27,
 abstract = {We present an empirical study of designing a NIME for the head-mounted augmented reality (HMAR) environment. In the NIME community, various sonic applications have incorporated augmented reality (AR) for sonic experience and audio production. With this novel digital form, new opportunities for musical expression and interface are presented. Yet few works consider whether and how the design of the NIME will be affected given the technology’s affordance. In this paper, we take an autobiographical design approach to design a NIME in HMAR, exploring what is a genuine application of AR in a NIMEs and how AR mediates between the performer and sound as a creative expression. Three interface prototypes are created for a frequency modulation synthesis system. We report on their design process and our learning and experiences through self-usage and improvisation. Our designs explore free-hand and embodied interaction in our interfaces, and we reflect on how these unique qualities of HMAR contribute to an expressive medium for sonic creation.},
 address = {The University of Auckland, New Zealand},
 articleno = {27},
 author = {Wang, Yichen and Martin, Charles},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.21428/92fbeb44.b540aa59},
 issn = {2220-4806},
 month = {jun},
 pdf = {183.pdf},
 presentation-video = {https://youtu.be/iOuZqwIwinU},
 title = {Cubing Sound: Designing a {NIME} for Head-mounted Augmented Reality},
 url = {https://doi.org/10.21428%2F92fbeb44.b540aa59},
 year = {2022}
}