Non-invasive sensing and gesture control for pitched percussion hyper-instruments using the Kinect

Shawn Trail, Michael Dean, Gabrielle Odowichuk, Tiago Fernandes Tavares, Peter Driessen, W. Andrew Schloss, and George Tzanetakis

Proceedings of the International Conference on New Interfaces for Musical Expression

• Year: 2012
• Location: Ann Arbor, Michigan
• DOI: 10.5281/zenodo.1178435 (Link to paper)
• PDF link

Abstract:

Hyper-instruments extend traditional acoustic instruments with sensing technologies that capture digitally subtle and sophisticated aspects of human performance. They leverage the long training and skills of performers while simultaneously providing rich possibilities for digital control. Many existing hyper-instruments suffer from being one of a kind instruments that require invasive modifications to the underlying acoustic instrument. In this paper we focus on the pitched percussion family and describe a non-invasive sensing approach for extending them to hyper-instruments. Our primary concern is to retain the technical integrity of the acoustic instrument and sound production methods while being able to intuitively interface the computer. This is accomplished by utilizing the Kinect sensor to track the position of the mallets without any modification to the instrument which enables easy and cheap replication of the pro-posed hyper-instrument extensions. In addition we describe two approaches to higher-level gesture control that remove the need for additional control devices such as foot pedals and fader boxes that are frequently used in electro-acoustic performance. This gesture control integrates more organically with the natural flow of playing the instrument providing user selectable control over filter parameters, synthesis, sampling, sequencing, and improvisation using a commer-cially available low-cost sensing apparatus.

Citation:

BibTeX Entry:

  @inproceedings{Trail2012a,
 abstract = {Hyper-instruments extend traditional acoustic instruments with sensing technologies that capture digitally subtle and sophisticated aspects of human performance. They leverage the long training and skills of performers while simultaneously providing rich possibilities for digital control. Many existing hyper-instruments suffer from being one of a kind instruments that require invasive modifications to the underlying acoustic instrument. In this paper we focus on the pitched percussion family and describe a non-invasive sensing approach for extending them to hyper-instruments. Our primary concern is to retain the technical integrity of the acoustic instrument and sound production methods while being able to intuitively interface the computer. This is accomplished by utilizing the Kinect sensor to track the position of the mallets without any modification to the instrument which enables easy and cheap replication of the pro-posed hyper-instrument extensions. In addition we describe two approaches to higher-level gesture control that remove the need for additional control devices such as foot pedals and fader boxes that are frequently used in electro-acoustic performance. This gesture control integrates more organically with the natural flow of playing the instrument providing user selectable control over filter parameters, synthesis, sampling, sequencing, and improvisation using a commer-cially available low-cost sensing apparatus.},
 address = {Ann Arbor, Michigan},
 author = {Shawn Trail and Michael Dean and Gabrielle Odowichuk and Tiago Fernandes Tavares and Peter Driessen and W. Andrew Schloss and George Tzanetakis},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.1178435},
 issn = {2220-4806},
 publisher = {University of Michigan},
 title = {Non-invasive sensing and gesture control for pitched percussion hyper-instruments using the Kinect},
 url = {http://www.nime.org/proceedings/2012/nime2012_297.pdf},
 year = {2012}
}