An Interactive Musical Prediction System with Mixture Density Recurrent Neural Networks

Charles Patrick Martin, and Jim Torresen

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract:

This paper is about creating digital musical instruments where a predictive neural network model is integrated into the interactive system. Rather than predicting symbolic music (e.g., MIDI notes), we suggest that predicting future control data from the user and precise temporal information can lead to new and interesting interactive possibilities. We propose that a mixture density recurrent neural network (MDRNN) is an appropriate model for this task. The predictions can be used to fill-in control data when the user stops performing, or as a kind of filter on the user's own input. We present an interactive MDRNN prediction server that allows rapid prototyping of new NIMEs featuring predictive musical interaction by recording datasets, training MDRNN models, and experimenting with interaction modes. We illustrate our system with several example NIMEs applying this idea. Our evaluation shows that real-time predictive interaction is viable even on single-board computers and that small models are appropriate for small datasets.

Citation:

Charles Patrick Martin, and Jim Torresen. 2019. An Interactive Musical Prediction System with Mixture Density Recurrent Neural Networks. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.3672952

BibTeX Entry:

  @inproceedings{Martin2019,
 abstract = {This paper is about creating digital musical instruments where a predictive neural network model is integrated into the interactive system. Rather than predicting symbolic music (e.g., MIDI notes), we suggest that predicting future control data from the user and precise temporal information can lead to new and interesting interactive possibilities. We propose that a mixture density recurrent neural network (MDRNN) is an appropriate model for this task. The predictions can be used to fill-in control data when the user stops performing, or as a kind of filter on the user's own input. We present an interactive MDRNN prediction server that allows rapid prototyping of new NIMEs featuring predictive musical interaction by recording datasets, training MDRNN models, and experimenting with interaction modes. We illustrate our system with several example NIMEs applying this idea. Our evaluation shows that real-time predictive interaction is viable even on single-board computers and that small models are appropriate for small datasets.},
 address = {Porto Alegre, Brazil},
 author = {Charles Patrick Martin and Jim Torresen},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.3672952},
 editor = {Marcelo Queiroz and Anna Xambó Sedó},
 issn = {2220-4806},
 month = {June},
 pages = {260--265},
 publisher = {UFRGS},
 title = {An Interactive Musical Prediction System with Mixture Density Recurrent Neural Networks},
 url = {http://www.nime.org/proceedings/2019/nime2019_paper050.pdf},
 year = {2019}
}