Playing Together with a Semi-Automated Robotic Flute Using a Gesture Cue Detection System

Jaeran Choi, Juhan Nam, Hikari Kuriyama, and Gou Koutaki

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

This study presents a semi-automated robotic flute system that coordinates performance onset timing using a human performer’s gesture cues, and examines how such control influences experience in a human–robot ensemble. In the proposed system, the performer produces sound through breath while the robot actuates the flute’s keys via a servo-driven mechanism, establishing a shared-performance structure. A camera-based motion tracking system detects preparatory head gestures in real time and predicts intended onset timing using a gesture cue–onset ratio model. We compared three conditions: timer-based onset, gesture cue-based onset with visual feedback, and gesture cue-based onset without visual feedback. Quantitative measures assessed onset asynchrony, and qualitative measures examined perceived partnership, agency, leadership, and trust. Results indicate that gesture cue-based control enhances the sense of partnership and performer agency, while timer-based control yields higher timing stability. These findings suggest that gesture-driven semi-automated musical robots can shift perception from playback device to ensemble partner.

Citation

Jaeran Choi, Juhan Nam, Hikari Kuriyama, and Gou Koutaki. 2026. Playing Together with a Semi-Automated Robotic Flute Using a Gesture Cue Detection System. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.20784392 [PDF]

BibTeX Entry


@inproceedings{nime2026_127,
abstract = {This study presents a semi-automated robotic flute system that coordinates performance onset timing using a human performer’s gesture cues, and examines how such control influences experience in a human–robot ensemble. In the proposed system, the performer produces sound through breath while the robot actuates the flute’s keys via a servo-driven mechanism, establishing a shared-performance structure. A camera-based motion tracking system detects preparatory head gestures in real time and predicts intended onset timing using a gesture cue–onset ratio model. We compared three conditions: timer-based onset, gesture cue-based onset with visual feedback, and gesture cue-based onset without visual feedback. Quantitative measures assessed onset asynchrony, and qualitative measures examined perceived partnership, agency, leadership, and trust. Results indicate that gesture cue-based control enhances the sense of partnership and performer agency, while timer-based control yields higher timing stability. These findings suggest that gesture-driven semi-automated musical robots can shift perception from playback device to ensemble partner.},
address = {London, United Kingdom},
articleno = {127},
author = {Jaeran Choi and Juhan Nam and Hikari Kuriyama and Gou Koutaki},
booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
doi = {10.5281/zenodo.20784392},
editor = {Benedict Gaster and João Tragtenberg and Anna Xambó and Tom Mitchell},
issn = {2220-4806},
month = {June},
numpages = {7},
pages = {1035--1041},
title = {Playing Together with a Semi-Automated Robotic Flute Using a Gesture Cue Detection System},
track = {paper},
url = {http://nime.org/proceedings/2026/nime2026_127.pdf},
year = {2026}
}