The Rearranger Ball: Delayed Gestural Control of Musical Sound using Online Unsupervised Temporal Segmentation

Juan Ignacio Mendoza Garay

Proceedings of the International Conference on New Interfaces for Musical Expression

• Year: 2023
• Location: Mexico City, Mexico
• Track: Papers
• Pages: 142–146
• Article Number: 19
• DOI: 10.5281/zenodo.11189133 (Link to paper)
• PDF link

Abstract:

The state-of-the-art recognition of continuous gestures for control of musical sound by means of machine learning has two notable constraints. The first is that the system needs to be trained with individual example gestures, the starting and ending points of which need to be well defined. The second constraint is time required for the system to recognise that a gesture has occurred, which may prevent the quick action that musical performance typically requires. This article describes how a method for unsupervised segmentation of gestures, may be used for delayed gestural control of a musical system. The system allows a user to perform without explicitly indicating the starting and ending of gestures in order to train the machine learning algorithm. To demonstrate the feasibility of the system, an apparatus for control of musical sound was devised incorporating the time required by the process into the interaction paradigm. The unsupervised automatic segmentation method and the concept of delayed control are further proposed to be exploited in the design and implementation of systems that facilitate seamless human-machine musical interaction without the need for quick response time, for example when using broad motion of the human body.

Citation:

BibTeX Entry:

  @inproceedings{nime2023_19,
 abstract = {The state-of-the-art recognition of continuous gestures for control of musical sound by means of machine learning has two notable constraints. The first is that the system needs to be trained with individual example gestures, the starting and ending points of which need to be well defined. The second constraint is time required for the system to recognise that a gesture has occurred, which may prevent the quick action that musical performance typically requires. This article describes how a method for unsupervised segmentation of gestures, may be used for delayed gestural control of a musical system. The system allows a user to perform without explicitly indicating the starting and ending of gestures in order to train the machine learning algorithm. To demonstrate the feasibility of the system, an apparatus for control of musical sound was devised incorporating the time required by the process into the interaction paradigm. The unsupervised automatic segmentation method and the concept of delayed control are further proposed to be exploited in the design and implementation of systems that facilitate seamless human-machine musical interaction without the need for quick response time, for example when using broad motion of the human body.},
 address = {Mexico City, Mexico},
 articleno = {19},
 author = {Juan Ignacio Mendoza Garay},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.11189133},
 editor = {Miguel Ortiz and Adnan Marquez-Borbon},
 issn = {2220-4806},
 month = {May},
 numpages = {5},
 pages = {142--146},
 title = {The Rearranger Ball: Delayed Gestural Control of Musical Sound using Online Unsupervised Temporal Segmentation},
 track = {Papers},
 url = {http://nime.org/proceedings/2023/nime2023_19.pdf},
 year = {2023}
}