Latent Mappings: Generating Open-Ended Expressive Mappings Using Variational Autoencoders

Tim Murray-Browne, and Panagiotis Tigas

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

In many contexts, creating mappings for gestural interactions can form part of an artistic process. Creators seeking a mapping that is expressive, novel, and affords them a sense of authorship may not know how to program it up in a signal processing patch. Tools like Wekinator [1] and MIMIC [2] allow creators to use supervised machine learning to learn mappings from example input/output pairings. However, a creator may know a good mapping when they encounter it yet start with little sense of what the inputs or outputs should be. We call this an open-ended mapping process. Addressing this need, we introduce the latent mapping, which leverages the latent space of an unsupervised machine learning algorithm such as a Variational Autoencoder trained on a corpus of unlabelled gestural data from the creator. We illustrate it with Sonified Body, a system mapping full-body movement to sound which we explore in a residency with three dancers.

Citation

Tim Murray-Browne, and Panagiotis Tigas. 2021. Latent Mappings: Generating Open-Ended Expressive Mappings Using Variational Autoencoders. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.21428/92fbeb44.9d4bcd4b

BibTeX Entry

@inproceedings{NIME21_66,
 abstract = {In many contexts, creating mappings for gestural interactions can form part of an artistic process. Creators seeking a mapping that is expressive, novel, and affords them a sense of authorship may not know how to program it up in a signal processing patch. Tools like Wekinator [1] and MIMIC [2] allow creators to use supervised machine learning to learn mappings from example input/output pairings. However, a creator may know a good mapping when they encounter it yet start with little sense of what the inputs or outputs should be. We call this an open-ended mapping process. Addressing this need, we introduce the latent mapping, which leverages the latent space of an unsupervised machine learning algorithm such as a Variational Autoencoder trained on a corpus of unlabelled gestural data from the creator. We illustrate it with Sonified Body, a system mapping full-body movement to sound which we explore in a residency with three dancers.},
 address = {Shanghai, China},
 articleno = {66},
 author = {Murray-Browne, Tim and Tigas, Panagiotis},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.21428/92fbeb44.9d4bcd4b},
 issn = {2220-4806},
 month = {June},
 presentation-video = {https://youtu.be/zBOHWyIGaYc},
 title = {Latent Mappings: Generating Open-Ended Expressive Mappings Using Variational Autoencoders},
 url = {https://nime.pubpub.org/pub/latent-mappings},
 year = {2021}
}