Playing Together with a Semi-Automated Robotic Flute Using a Gesture Cue Detection System

Jaeran Choi, Juhan Nam, Hikari Kuriyama, and Gou Koutaki

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

This study presents a semi-automated robotic flute system that coordinates performance onset timing using a human performer’s gesture cues, and examines how such control influences experience in a human–robot ensemble. In the proposed system, the performer produces sound through breath while the robot actuates the flute’s keys via a servo-driven mechanism, establishing a shared-performance structure. A camera-based motion tracking system detects preparatory head gestures in real time and predicts intended onset timing using a gesture cue–onset ratio model. We compared three conditions: timer-based onset, gesture cue-based onset with visual feedback, and gesture cue-based onset without visual feedback. Quantitative measures assessed onset asynchrony, and qualitative measures examined perceived partnership, agency, leadership, and trust. Results indicate that gesture cue-based control enhances the sense of partnership and performer agency, while timer-based control yields higher timing stability. These findings suggest that gesture-driven semi-automated musical robots can shift perception from playback device to ensemble partner.

Citation

Jaeran Choi, Juhan Nam, Hikari Kuriyama, and Gou Koutaki. 2026. Playing Together with a Semi-Automated Robotic Flute Using a Gesture Cue Detection System. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.20784392 [PDF]

BibTeX Entry

@inproceedings{nime2026_127,
 abstract = {This study presents a semi-automated robotic flute system that coordinates performance onset timing using a human performer’s gesture cues, and examines how such control influences experience in a human–robot ensemble. In the proposed system, the performer produces sound through breath while the robot actuates the flute’s keys via a servo-driven mechanism, establishing a shared-performance structure. A camera-based motion tracking system detects preparatory head gestures in real time and predicts intended onset timing using a gesture cue–onset ratio model. We compared three conditions: timer-based onset, gesture cue-based onset with visual feedback, and gesture cue-based onset without visual feedback. Quantitative measures assessed onset asynchrony, and qualitative measures examined perceived partnership, agency, leadership, and trust. Results indicate that gesture cue-based control enhances the sense of partnership and performer agency, while timer-based control yields higher timing stability. These findings suggest that gesture-driven semi-automated musical robots can shift perception from playback device to ensemble partner.},
 address = {London, United Kingdom},
 articleno = {127},
 author = {Jaeran Choi and Juhan Nam and Hikari Kuriyama and Gou Koutaki},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.20784392},
 editor = {Benedict Gaster and João Tragtenberg and Anna Xambó and Tom Mitchell},
 issn = {2220-4806},
 month = {June},
 note = {},
 numpages = {7},
 pages = {1035--1041},
 title = {Playing Together with a Semi-Automated Robotic Flute Using a Gesture Cue Detection System},
 track = {paper},
 url = {http://nime.org/proceedings/2026/nime2026_127.pdf},
 year = {2026}
}