Skip to content

Intuitive Real-Time Control of Spectral Model Synthesis

Phillip Popp, and Matthew Wright

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

Several methods exist for manipulating spectral models either by applying transformations via higher level features or by providing in-depth offline editing capabilities. In contrast, our system aims for direct, full, intuitive, real-time control without exposing any spectral model features to the user. The system extends upon previous machine learning work in gesture-synthesis mapping by applying it to spectral models; these are a unique and interesting use case in that they are capable of reproducing real world recordings, due to their relatively high data rate and complex, intertwined and synergetic structure. To achieve a direct and intuitive control of a spectral model, a method to extract an individualized mapping between Wacom Pen parameters and Spectral Model Synthesis frames is described and implemented as a standalone application. The method works by capturing tablet parameters as the user pantomimes to synthesized spectral model. A transformation from Wacom Pen parameters to gestures is obtained by extracting features from the pen and then transforming those features using Principal Component Analysis. Then a linear model maps between gestures and higher level features of the spectral model frames while a k-nearest neighbor algorithm maps between gestures and normalized spectral model frames.

Citation

Phillip Popp, and Matthew Wright. 2011. Intuitive Real-Time Control of Spectral Model Synthesis. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.1178139 [PDF]

BibTeX Entry


@inproceedings{nime2011_Popp,
author = {Popp, Phillip and Wright, Matthew},
title = {Intuitive Real-Time Control of Spectral Model Synthesis},
pages = {284--287},
booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
year = {2011},
address = {Oslo, Norway},
issn = {2220-4806},
doi = {10.5281/zenodo.1178139},
url = {https://nime.org/proceedings/2011/nime2011_284.pdf},
keywords = {Spectral Model Synthesis, Gesture Recognition, Synthesis Control, Wacom Tablet, Machine Learning },
abstract = {Several methods exist for manipulating spectral models either by applying transformations via higher level features or by providing in-depth offline editing capabilities. In contrast, our system aims for direct, full, intuitive, real-time control without exposing any spectral model features to the user. The system extends upon previous machine learning work in gesture-synthesis mapping by applying it to spectral models; these are a unique and interesting use case in that they are capable of reproducing real world recordings, due to their relatively high data rate and complex, intertwined and synergetic structure. To achieve a direct and intuitive control of a spectral model, a method to extract an individualized mapping between Wacom Pen parameters and Spectral Model Synthesis frames is described and implemented as a standalone application. The method works by capturing tablet parameters as the user pantomimes to synthesized spectral model. A transformation from Wacom Pen parameters to gestures is obtained by extracting features from the pen and then transforming those features using Principal Component Analysis. Then a linear model maps between gestures and higher level features of the spectral model frames while a k-nearest neighbor algorithm maps between gestures and normalized spectral model frames.}
}