Sonification of Facial Actions for Musical Expression

Mathias Funk, Kazuhiro Kuwabara, and Michael J. Lyons

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract:

The central role of the face in social interaction and non-verbal communication suggest we explore facial action as a means of musical expression. This paper presents the design, implementation, and preliminary studies of a novel system utilizing face detection and optic flow algorithms to associate facial movements with sound synthesis in a topographically specific fashion. We report on our experience with various gesture-to-sound mappings and applications, and describe our preliminary experiments at musical performance using the system.

Citation:

Mathias Funk, Kazuhiro Kuwabara, and Michael J. Lyons. 2005. Sonification of Facial Actions for Musical Expression. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.1176750

BibTeX Entry:

  @inproceedings{Funk2005,
 abstract = {The central role of the face in social interaction and non-verbal communication suggest we explore facial action as a means of musical expression. This paper presents the design, implementation, and preliminary studies of a novel system utilizing face detection and optic flow algorithms to associate facial movements with sound synthesis in a topographically specific fashion. We report on our experience with various gesture-to-sound mappings and applications, and describe our preliminary experiments at musical performance using the system. },
 address = {Vancouver, BC, Canada},
 author = {Funk, Mathias and Kuwabara, Kazuhiro and Lyons, Michael J.},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.1176750},
 issn = {2220-4806},
 keywords = {Video-based musical interface; gesture-based interaction; facial expression; facial therapy interface. },
 pages = {127--131},
 title = {Sonification of Facial Actions for Musical Expression},
 url = {http://www.nime.org/proceedings/2005/nime2005_127.pdf},
 year = {2005}
}