Enhancing Expressive Musical Conversation in the jam_bot

Lancelot Blanchard, Perry Naseck, Katherine Liang, Joel Tan, Heidi Lei, Cheng-Zhi Anna Huang, and Joseph Paradiso

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

Previous work introduced the jam_bot, a real-time system that embeds live music language models capable of generating symbolic music sequences coherent with a performer’s input. The system supports multiple interaction strategies that have been demonstrated in several public performances. However, these strategies limit expressive musical conversation by constraining tempo, form, or musical roles. We extend the jam_bot to support more expressive, open-ended interaction through four key improvements: (1) modeling velocity, a key dimension of expression in symbolic music; (2) increasing model throughput via a ggml implementation–required to accommodate the longer sequences induced by velocity modeling; (3) developing a new training modality that enables free-form call-and-response interaction across varying tempi; and (4) compensating for external MIDI output latency to ensure rhythmic coherence with the performer. We quantitatively evaluate the model throughput improvement and our latency compensation strategy, and offer MIDI samples online. Together, these enhancements enable the jam_bot to engage in natural, expressive musical conversation, eliminating key musical limitations to enable the development of future performances and installations.

Citation

Lancelot Blanchard, Perry Naseck, Katherine Liang, Joel Tan, Heidi Lei, Cheng-Zhi Anna Huang, and Joseph Paradiso. 2026. Enhancing Expressive Musical Conversation in the jam_bot. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.20784228 [PDF]

BibTeX Entry

@inproceedings{nime2026_73,
 abstract = {Previous work introduced the jam_bot, a real-time system that embeds live music language models capable of generating symbolic music sequences coherent with a performer’s input. The system supports multiple interaction strategies that have been demonstrated in several public performances. However, these strategies limit expressive musical conversation by constraining tempo, form, or musical roles. We extend the jam_bot to support more expressive, open-ended interaction through four key improvements: (1) modeling velocity, a key dimension of expression in symbolic music; (2) increasing model throughput via a ggml implementation–required to accommodate the longer sequences induced by velocity modeling; (3) developing a new training modality that enables free-form call-and-response interaction across varying tempi; and (4) compensating for external MIDI output latency to ensure rhythmic coherence with the performer. We quantitatively evaluate the model throughput improvement and our latency compensation strategy, and offer MIDI samples online. Together, these enhancements enable the jam_bot to engage in natural, expressive musical conversation, eliminating key musical limitations to enable the development of future performances and installations.},
 address = {London, United Kingdom},
 articleno = {73},
 author = {Lancelot Blanchard and Perry Naseck and Katherine Liang and Joel Tan and Heidi Lei and Cheng-Zhi Anna Huang and Joseph Paradiso},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.20784228},
 editor = {Benedict Gaster and João Tragtenberg and Anna Xambó and Tom Mitchell},
 issn = {2220-4806},
 month = {June},
 note = {},
 numpages = {6},
 pages = {621--626},
 presentation-video = {https://youtu.be/KpVISrUV408},
 title = {Enhancing Expressive Musical Conversation in the jam_bot},
 track = {Paper},
 url = {http://nime.org/proceedings/2026/nime2026_73.pdf},
 urlsuppl1 = {http://nime.org/proceedings/2026/nime2026_73_file01.mp4},
 year = {2026}
}