SnakeSynth: New Interactions for Generative Audio Synthesis

Eric Easthope

Proceedings of the International Conference on New Interfaces for Musical Expression

  • Year: 2023
  • Location: Mexico City, Mexico
  • Track: Work in Progress
  • Pages: 612–619
  • Article Number: 90
  • PDF link

Abstract:

We present SnakeSynth, a web-based lightweight audio synthesizer that combines audio generated by a deep generative model and real-time continuous two-dimensional (2D) input to create and control variable-length generative sounds through 2D interaction gestures. Interaction gestures are touch and mobile-compatible with analogies to strummed, bowed, and plucked musical instrument controls. Point-and-click and drag-and-drop gestures directly control audio playback length and we show that sound length and intensity are modulated by interactions with a programmable 2D coordinate grid. Leveraging the speed and ubiquity of browser-based audio and hardware acceleration in Google's "TensorFlow.js" we generate time-varying high-fidelity sounds with real-time interactivity. SnakeSynth adaptively reproduces and interpolates between sounds encountered during model training, notably without long training times, and we briefly discuss possible futures for deep generative models as an interactive paradigm for musical expression.

Citation:

Eric Easthope. 2023. SnakeSynth: New Interactions for Generative Audio Synthesis. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI:

BibTeX Entry:

  @article{nime2023_90,
 abstract = {We present SnakeSynth, a web-based lightweight audio synthesizer that combines audio generated by a deep generative model and real-time continuous two-dimensional (2D) input to create and control variable-length generative sounds through 2D interaction gestures. Interaction gestures are touch and mobile-compatible with analogies to strummed, bowed, and plucked musical instrument controls. Point-and-click and drag-and-drop gestures directly control audio playback length and we show that sound length and intensity are modulated by interactions with a programmable 2D coordinate grid. Leveraging the speed and ubiquity of browser-based audio and hardware acceleration in Google's "TensorFlow.js" we generate time-varying high-fidelity sounds with real-time interactivity. SnakeSynth adaptively reproduces and interpolates between sounds encountered during model training, notably without long training times, and we briefly discuss possible futures for deep generative models as an interactive paradigm for musical expression.},
 address = {Mexico City, Mexico},
 articleno = {90},
 author = {Eric Easthope},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 editor = {Miguel Ortiz and Adnan Marquez-Borbon},
 issn = {2220-4806},
 month = {May},
 numpages = {8},
 pages = {612--619},
 title = {SnakeSynth: New Interactions for Generative Audio Synthesis},
 track = {Work in Progress},
 url = {http://nime.org/proceedings/2023/nime2023_90.pdf},
 year = {2023}
}