GestureRNN: A neural gesture system for the Roli Lightpad Block

Lamtharn Hantrakul

Proceedings of the International Conference on New Interfaces for Musical Expression

• Year: 2018
• Location: Blacksburg, Virginia, USA
• Pages: 132–137
• DOI: 10.5281/zenodo.1302703 (Link to paper)
• PDF link

Abstract:

Machine learning and deep learning has recently made a large impact in the artistic community. In many of these applications however, the model is often used to render the high dimensional output directly e.g. every individual pixel in the final image. Humans arguably operate in much lower dimensional spaces during the creative process e.g. the broad movements of a brush. In this paper, we design a neural gesture system for music generation based around this concept. Instead of directly generating audio, we train a Long Short Term Memory (LSTM) recurrent neural network to generate instantaneous position and pressure on the Roli Lightpad instrument. These generated coordinates in turn, give rise to the sonic output defined in the synth engine. The system relies on learning these movements from a musician who has already developed a palette of musical gestures idiomatic to the Lightpad. Unlike many deep learning systems that render high dimensional output, our low-dimensional system can be run in real-time, enabling the first real time gestural duet of its kind between a player and a recurrent neural network on the Lightpad instrument.

Citation:

BibTeX Entry:

  @inproceedings{Hantrakul2018,
 abstract = {Machine learning and deep learning has recently made a large impact in the artistic community. In many of these applications however, the model is often used to render the high dimensional output directly e.g. every individual pixel in the final image. Humans arguably operate in much lower dimensional spaces during the creative process e.g. the broad movements of a brush. In this paper, we design a neural gesture system for music generation based around this concept. Instead of directly generating audio, we train a Long Short Term Memory (LSTM) recurrent neural network to generate instantaneous position and pressure on the Roli Lightpad instrument. These generated coordinates in turn, give rise to the sonic output defined in the synth engine.  The system relies on learning these movements from a musician who has already developed a palette of musical gestures idiomatic to the Lightpad. Unlike many deep learning systems that render high dimensional output, our low-dimensional system can be run in real-time, enabling the first real time gestural duet of its kind between a player and a recurrent neural network on the Lightpad instrument.},
 address = {Blacksburg, Virginia, USA},
 author = {Lamtharn Hantrakul},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.1302703},
 editor = {Luke Dahl, Douglas Bowman, Thomas Martin},
 isbn = {978-1-949373-99-8},
 issn = {2220-4806},
 month = {June},
 pages = {132--137},
 publisher = {Virginia Tech},
 title = {GestureRNN:  A neural gesture system for the Roli Lightpad Block},
 url = {http://www.nime.org/proceedings/2018/nime2018_paper0029.pdf},
 year = {2018}
}