MuGeVI: A Multi-Functional Gesture-Controlled Virtual Instrument

Yue Yang, Zhaowen Wang, and Zijin Li

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract:

Currently, most of the digital musical instruments cannot leave the use of dedicated hardware devices, making them limited in terms of user popularity and resource conservation. In this paper, we propose a new computer vision-based interactive multi-functional musical instrument, called MuGeVI, which requires no additional hardware circuits or sensors, and allows users to create or play music through different hand gestures and positions. It firstly uses deep neural network models for hand key point detection to obtain gesture information, secondly maps it to pitch, chord or other information based on the current mode, then passes it to Max/MSP via the OSC protocol, and finally implements the generation and processing of MIDI or audio. MuGeVI is now available in four modes: performance mode, accompaniment mode, control mode, and audio effects mode, and can be conveniently used with just a personal computer with a camera. Designed to be human-centric, MuGeVI is feature-rich, simple to use, affordable, scalable and programmable, and is certainly a frugal musical innovation. All the material about this work can be found in https://yewlife.github.io/MuGeVI/.

Citation:

Yue Yang, Zhaowen Wang, and Zijin Li. 2023. MuGeVI: A Multi-Functional Gesture-Controlled Virtual Instrument. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.11189278

BibTeX Entry:

  @inproceedings{nime2023_75,
 abstract = {Currently, most of the digital musical instruments cannot leave the use of dedicated hardware devices, making them limited in terms of user popularity and resource conservation. In this paper, we propose a new computer vision-based interactive multi-functional musical instrument, called MuGeVI, which requires no additional hardware circuits or sensors, and allows users to create or play music through different hand gestures and positions. It firstly uses deep neural network models for hand key point detection to obtain gesture information, secondly maps it to pitch, chord or other information based on the current mode, then passes it to Max/MSP via the OSC protocol, and finally implements the generation and processing of MIDI or audio. MuGeVI is now available in four modes: performance mode, accompaniment mode, control mode, and audio effects mode, and can be conveniently used with just a personal computer with a camera. Designed to be human-centric, MuGeVI is feature-rich, simple to use, affordable, scalable and programmable, and is certainly a frugal musical innovation. All the material about this work can be found in https://yewlife.github.io/MuGeVI/.},
 address = {Mexico City, Mexico},
 articleno = {75},
 author = {Yue Yang and Zhaowen Wang and Zijin Li},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.11189278},
 editor = {Miguel Ortiz and Adnan Marquez-Borbon},
 issn = {2220-4806},
 month = {May},
 numpages = {6},
 pages = {536--541},
 title = {MuGeVI: A Multi-Functional Gesture-Controlled Virtual Instrument},
 track = {Work in Progress},
 url = {http://nime.org/proceedings/2023/nime2023_75.pdf},
 year = {2023}
}