What to Play and How to Play it: Guiding Generative Music Models with Multiple Demonstrations

Jon Gillick, and David Bamman

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract:

We propose and evaluate an approach to incorporating multiple user-provided inputs, each demonstrating a complementary set of musical characteristics, to guide the output of a generative model for synthesizing short music performances or loops. We focus on user inputs that describe both “what to play” (via scores in MIDI format) and “how to play it” (via rhythmic inputs to specify expressive timing and dynamics). Through experiments, we demonstrate that our method can facilitate human-AI co-creation of drum loops with diverse and customizable outputs. In the process, we argue for the interaction paradigm of mapping by demonstration as a promising approach to working with deep learning models that are capable of generating complex and realistic musical parts.

Citation:

Jon Gillick, and David Bamman. 2021. What to Play and How to Play it: Guiding Generative Music Models with Multiple Demonstrations. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.21428/92fbeb44.06e2d5f4

BibTeX Entry:

  @inproceedings{NIME21_6,
 abstract = {We propose and evaluate an approach to incorporating multiple user-provided inputs, each demonstrating a complementary set of musical characteristics, to guide the output of a generative model for synthesizing short music performances or loops. We focus on user inputs that describe both “what to play” (via scores in MIDI format) and “how to play it” (via rhythmic inputs to specify expressive timing and dynamics). Through experiments, we demonstrate that our method can facilitate human-AI co-creation of drum loops with diverse and customizable outputs. In the process, we argue for the interaction paradigm of mapping by demonstration as a promising approach to working with deep learning models that are capable of generating complex and realistic musical parts.},
 address = {Shanghai, China},
 articleno = {6},
 author = {Gillick, Jon and Bamman, David},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.21428/92fbeb44.06e2d5f4},
 issn = {2220-4806},
 month = {June},
 presentation-video = {https://youtu.be/Q2M_smiN6oo},
 title = {What to Play and How to Play it: Guiding Generative Music Models with Multiple Demonstrations},
 url = {https://nime.pubpub.org/pub/s3x60926},
 year = {2021}
}