Designing, Playing, and Performing with a Vision-based Mouth Interface

Michael J. Lyons, Michael Haehnel, and Nobuji Tetsutani

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

The role of the face and mouth in speech production as well asnon-verbal communication suggests the use of facial action tocontrol musical sound. Here we document work on theMouthesizer, a system which uses a headworn miniaturecamera and computer vision algorithm to extract shapeparameters from the mouth opening and output these as MIDIcontrol changes. We report our experience with variousgesture-to-sound mappings and musical applications, anddescribe a live performance which used the Mouthesizerinterface.

Citation

Michael J. Lyons, Michael Haehnel, and Nobuji Tetsutani. 2003. Designing, Playing, and Performing with a Vision-based Mouth Interface. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.1176529 [PDF]

BibTeX Entry

@inproceedings{nime2003_Lyons,
 abstract = {The role of the face and mouth in speech production as well asnon-verbal communication suggests the use of facial action tocontrol musical sound. Here we document work on theMouthesizer, a system which uses a headworn miniaturecamera and computer vision algorithm to extract shapeparameters from the mouth opening and output these as MIDIcontrol changes. We report our experience with variousgesture-to-sound mappings and musical applications, anddescribe a live performance which used the Mouthesizerinterface.},
 address = {Montreal, Canada},
 author = {Lyons, Michael J. and Haehnel, Michael and Tetsutani, Nobuji},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 date = {22-24 May, 2003},
 doi = {10.5281/zenodo.1176529},
 issn = {2220-4806},
 keywords = {Video-based interface; mouth controller; alternative input devices. },
 pages = {116--121},
 title = {Designing, Playing, and Performing with a Vision-based Mouth Interface},
 url = {http://www.nime.org/proceedings/2003/nime2003_116.pdf},
 year = {2003}
}