Interface Modules for Extended Reality in Music

Damian Dziwis, and Aristotelis Hadjakos

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract:

The growing possibilities and availability of immersive technologies in the field of Extended Reality (XR) are leading to new potentials in the context of Musical XR, like the emergence of new concepts for virtual reality musical instruments (VRMIs), reception, and composition and performance practices. Recent XR devices for Virtual and Augmented Reality allow not only to present virtual content, but also to blend real and virtual environments to create hybrid Mixed Realities. Therefore, in addition to using standard interfaces such as controllers or hand tracking from XR systems for interacting with VRMIs, developing custom musical interfaces can provide enhanced physical possibilities and experiences for interacting with VRMIs and other virtual sound-generating systems. This paper describes the development and possibilities of an open-source modular interface system for the realization of VRMIs with physical musical interfaces, interactive installations and the augmentation of virtual environments with physical environmental information.The tool was created by integrating the SPINE sensor-based musical interface toolkit with the IVES modular 3D engine for the Max programming environment. Merging these systems, with their no-code/no-soldering approach, results in a customizable tool for rapidly creating VRMIs or augmented virtual environments with corresponding physical interfaces.

Citation:

Damian Dziwis, and Aristotelis Hadjakos. 2024. Interface Modules for Extended Reality in Music. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.13904903

BibTeX Entry:

  @article{nime2024_64,
 abstract = {The growing possibilities and availability of immersive technologies in the field of Extended Reality (XR) are leading to new potentials in the context of Musical XR, like the emergence of new concepts for virtual reality musical instruments (VRMIs), reception, and composition and performance practices. Recent XR devices for Virtual and Augmented Reality allow not only to present virtual content, but also to blend real and virtual environments to create hybrid Mixed Realities. Therefore, in addition to using standard interfaces such as controllers or hand tracking from XR systems for interacting with VRMIs, developing custom musical interfaces can provide enhanced physical possibilities and experiences for interacting with VRMIs and other virtual sound-generating systems. This paper describes the development and possibilities of an open-source modular interface system for the realization of VRMIs with physical musical interfaces, interactive installations and the augmentation of virtual environments with physical environmental information.The tool was created by integrating the SPINE sensor-based musical interface toolkit with the IVES modular 3D engine for the Max programming environment. Merging these systems, with their no-code/no-soldering approach, results in a customizable tool for rapidly creating VRMIs or augmented virtual environments with corresponding physical interfaces.},
 address = {Utrecht, Netherlands},
 articleno = {64},
 author = {Damian Dziwis and Aristotelis Hadjakos},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.13904903},
 editor = {S M Astrid Bin and Courtney N. Reed},
 issn = {2220-4806},
 month = {September},
 numpages = {8},
 pages = {439--446},
 presentation-video = {},
 title = {Interface Modules for Extended Reality in Music},
 track = {Papers},
 url = {http://nime.org/proceedings/2024/nime2024_64.pdf},
 year = {2024}
}