Inspecting and Interacting with Meaningful Music Representations using VAE

Ruihan Yang, Tianyao Chen, Yiyi Zhang, and gus xia

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract:

Variational Autoencoder has already achieved great results on image generation and recently made promising progress on music sequence generation. However, the model is still quite difficult to control in the sense that the learned latent representations lack meaningful music semantics. What users really need is to interact with certain music features, such as rhythm and pitch contour, in the creation process so that they can easily test different composition ideas. In this paper, we propose a disentanglement by augmentation method to inspect the pitch and rhythm interpretations of the latent representations. Based on the interpretable representations, an intuitive graphical user interface demo is designed for users to better direct the music creation process by manipulating the pitch contours and rhythmic complexity.

Citation:

Ruihan Yang, Tianyao Chen, Yiyi Zhang, and gus xia. 2019. Inspecting and Interacting with Meaningful Music Representations using VAE. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.3672974

BibTeX Entry:

  @inproceedings{Yang2019,
 abstract = {Variational Autoencoder has already achieved great results on image generation  and recently made promising progress on music sequence generation. However, the model is still quite difficult to  control in the sense that the learned latent representations lack meaningful music semantics. What users really need is to interact with certain  music features, such as rhythm and pitch contour, in the creation process so that they can easily test different composition ideas. In this paper, we propose a disentanglement by augmentation method to inspect the pitch and rhythm interpretations of the latent representations. Based on the interpretable representations, an intuitive graphical user interface demo is designed for users to better direct the music creation process by manipulating the pitch contours and rhythmic complexity.},
 address = {Porto Alegre, Brazil},
 author = {Ruihan Yang and Tianyao Chen and Yiyi Zhang and gus xia},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.3672974},
 editor = {Marcelo Queiroz and Anna Xambó Sedó},
 issn = {2220-4806},
 month = {June},
 pages = {307--312},
 publisher = {UFRGS},
 title = {Inspecting and Interacting with Meaningful Music Representations using {VAE}},
 url = {http://www.nime.org/proceedings/2019/nime2019_paper059.pdf},
 year = {2019}
}