Live Improvisation with Fine-Tuned Generative AI: A Musical Metacreation Approach

Misagh Azimi, and Mo H. Zareei

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

This paper presents a pipeline to integrate a fine-tuned open-source text-to-audio latent diffusion model into a workflow with Ableton Live for the improvisation of contemporary electronic music. The system generates audio fragments based on text prompts provided in real time by the performer, enabling dynamic interaction. Guided by Musical Metacreation as a framework, this case study reframes generative AI as a co-creative agent rather than a mere style imitator. By fine-tuning Stable Audio Open on a dataset of the first author’s compositions and field recordings, this approach demonstrates the ethical and practical benefits of open-source solutions. Beyond showcasing the model’s creative potential, this study highlights the model’s significant challenges and the need for democratized tools with real-world applications.

Citation

Misagh Azimi, and Mo H. Zareei. 2025. Live Improvisation with Fine-Tuned Generative AI: A Musical Metacreation Approach. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.15698902 [PDF]

BibTeX Entry

@article{nime2025_54,
 abstract = {This paper presents a pipeline to integrate a fine-tuned open-source text-to-audio latent diffusion model into a workflow with Ableton Live for the improvisation of contemporary electronic music. The system generates audio fragments based on text prompts provided in real time by the performer, enabling dynamic interaction. Guided by Musical Metacreation as a framework, this case study reframes generative AI as a co-creative agent rather than a mere style imitator. By fine-tuning Stable Audio Open on a dataset of the first author’s compositions and field recordings, this approach demonstrates the ethical and practical benefits of open-source solutions. Beyond showcasing the model’s creative potential, this study highlights the model’s significant challenges and the need for democratized tools with real-world applications.},
 address = {Canberra, Australia},
 articleno = {54},
 author = {Misagh Azimi and Mo H. Zareei},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.15698902},
 editor = {Doga Cavdir and Florent Berthaut},
 issn = {2220-4806},
 month = {June},
 numpages = {5},
 pages = {389--393},
 title = {Live Improvisation with Fine-Tuned Generative AI: A Musical Metacreation Approach},
 track = {Paper},
 url = {http://nime.org/proceedings/2025/nime2025_54.pdf},
 year = {2025}
}