Calliphony: A Calligraphy-Driven Interface for Real-Time Generative Music Performance

Tristan WU, Ruiji YU, and Gus XIA

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

While music generative models have recently gained significant attention, how they can be effectively integrated into live music performances still requires further exploration. This paper presents Calliphony, a calligraphy-driven interface for real-time generative music performance. Specifically, we build a low-latency pipeline that captures brush motion with an attachable sensor and maps it to control signals for real-time symbolic music generation. Using a generative model, the system produces multi-track MIDI in performance settings, while brush-derived control signals constrain event timing and activate additional musical layers.The generated melody is then extended with real-time harmony and additional voices, and finally rendered through a DAW for live staging.Calliphony contributes: (1) a performance-oriented prototype that uses calligraphic motion as an external control layer for a real-time symbolic music generation model, controlling note density, pitch constraints, and accompaniment-layer activation; and (2) a cross-modal performance scenario that extends calligraphy beyond a primarily visual practice into an audiovisual, AI-assisted setting.

Citation

Tristan WU, Ruiji YU, and Gus XIA. 2026. Calliphony: A Calligraphy-Driven Interface for Real-Time Generative Music Performance. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.20784121 [PDF]

BibTeX Entry

@inproceedings{nime2026_32,
 abstract = {While music generative models have recently gained significant attention, how they can be effectively integrated into live music performances still requires further exploration. This paper presents Calliphony, a calligraphy-driven interface for real-time generative music performance. Specifically, we build a low-latency pipeline that captures brush motion with an attachable sensor and maps it to control signals for real-time symbolic music generation. Using a generative model, the system produces multi-track MIDI in performance settings, while brush-derived control signals constrain event timing and activate additional musical layers.The generated melody is then extended with real-time harmony and additional voices, and finally rendered through a DAW for live staging.Calliphony contributes: (1) a performance-oriented prototype that uses calligraphic motion as an external control layer for a real-time symbolic music generation model, controlling note density, pitch constraints, and accompaniment-layer activation; and (2) a cross-modal performance scenario that extends calligraphy beyond a primarily visual practice into an audiovisual, AI-assisted setting.},
 address = {London, United Kingdom},
 articleno = {32},
 author = {Tristan WU and Ruiji YU and Gus XIA},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.20784121},
 editor = {Benedict Gaster and João Tragtenberg and Anna Xambó and Tom Mitchell},
 issn = {2220-4806},
 month = {June},
 note = {},
 numpages = {6},
 pages = {270--275},
 presentation-video = {https://youtu.be/Pp3JmC51GYI},
 title = {Calliphony: A Calligraphy-Driven Interface for Real-Time Generative Music Performance},
 track = {Paper},
 url = {http://nime.org/proceedings/2026/nime2026_32.pdf},
 year = {2026}
}