Sonification of Facial Actions for Musical Expression

Mathias Funk, Kazuhiro Kuwabara, and Michael J. Lyons

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

The central role of the face in social interaction and non-verbal communication suggest we explore facial action as a means of musical expression. This paper presents the design, implementation, and preliminary studies of a novel system utilizing face detection and optic flow algorithms to associate facial movements with sound synthesis in a topographically specific fashion. We report on our experience with various gesture-to-sound mappings and applications, and describe our preliminary experiments at musical performance using the system.

Citation

Mathias Funk, Kazuhiro Kuwabara, and Michael J. Lyons. 2005. Sonification of Facial Actions for Musical Expression. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.1176750

BibTeX Entry

@inproceedings{Funk2005,
 abstract = {The central role of the face in social interaction and non-verbal communication suggest we explore facial action as a means of musical expression. This paper presents the design, implementation, and preliminary studies of a novel system utilizing face detection and optic flow algorithms to associate facial movements with sound synthesis in a topographically specific fashion. We report on our experience with various gesture-to-sound mappings and applications, and describe our preliminary experiments at musical performance using the system. },
 address = {Vancouver, BC, Canada},
 author = {Funk, Mathias and Kuwabara, Kazuhiro and Lyons, Michael J.},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.1176750},
 issn = {2220-4806},
 keywords = {Video-based musical interface; gesture-based interaction; facial expression; facial therapy interface. },
 pages = {127--131},
 title = {Sonification of Facial Actions for Musical Expression},
 url = {http://www.nime.org/proceedings/2005/nime2005_127.pdf},
 year = {2005}
}