Imagined Movement as Sonic Gesture: Auditory Expression from a Deep Learning-Based Motion Decoding BCI

Niall McShane, Karl McCreadie, Attila Korik, and Damien Coyle

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

Continuous motion trajectory decoding (MTD) brain-computer interfaces (BCIs) translate imagined limb movements into continuous control signals, traditionally presented through visual feedback such as virtual limbs. This paper extends the multimodal expressivity of MTD-BCIs by introducing embodied sonification as a primary interaction modality. Using previously trained CNN-LSTM decoders for three-dimensional imagined arm movement, we mapped decoded motion and velocity signals in real time to a layered granular synthesis system. The framework employs velocity magnitude to modulate textural density and spectral characteristics, temporal accumulation to shape harmonic evolution, and rest-state detection to define acoustic boundaries between imagined gesture phases. Rather than treating sound as supplementary feedback, this approach positions decoded motion as sonic gesture and a continuous expressive articulation of imagined movement grounded in embodied cognition principles. The multi-layered synthesis architecture demonstrates that complex, many-to-many parameter mappings preserve perceptual coherence between movement dynamics and auditory change, creating a unified motion-audio-visual loop. This proof-of-concept establishes sonification as a viable interaction paradigm for motion-based BCIs, with implications for expressive musical performance, accessible sound-based control, and neurocognitive research into multimodal feedback in embodied human-computer interaction.

Citation

Niall McShane, Karl McCreadie, Attila Korik, and Damien Coyle. 2026. Imagined Movement as Sonic Gesture: Auditory Expression from a Deep Learning-Based Motion Decoding BCI. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.20784295 [PDF]

BibTeX Entry

@inproceedings{nime2026_98,
 abstract = {Continuous motion trajectory decoding (MTD) brain-computer interfaces (BCIs) translate imagined limb movements into continuous control signals, traditionally presented through visual feedback such as virtual limbs. This paper extends the multimodal expressivity of MTD-BCIs by introducing embodied sonification as a primary interaction modality. Using previously trained CNN-LSTM decoders for three-dimensional imagined arm movement, we mapped decoded motion and velocity signals in real time to a layered granular synthesis system. The framework employs velocity magnitude to modulate textural density and spectral characteristics, temporal accumulation to shape harmonic evolution, and rest-state detection to define acoustic boundaries between imagined gesture phases. Rather than treating sound as supplementary feedback, this approach positions decoded motion as sonic gesture and a continuous expressive articulation of imagined movement grounded in embodied cognition principles. The multi-layered synthesis architecture demonstrates that complex, many-to-many parameter mappings preserve perceptual coherence between movement dynamics and auditory change, creating a unified motion-audio-visual loop. This proof-of-concept establishes sonification as a viable interaction paradigm for motion-based BCIs, with implications for expressive musical performance, accessible sound-based control, and neurocognitive research into multimodal feedback in embodied human-computer interaction.},
 address = {London, United Kingdom},
 articleno = {98},
 author = {Niall McShane and Karl McCreadie and Attila Korik and Damien Coyle},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.20784295},
 editor = {Benedict Gaster and João Tragtenberg and Anna Xambó and Tom Mitchell},
 issn = {2220-4806},
 month = {June},
 note = {},
 numpages = {9},
 pages = {832--840},
 title = {Imagined Movement as Sonic Gesture: Auditory Expression from a Deep Learning-Based Motion Decoding BCI},
 track = {paper},
 url = {http://nime.org/proceedings/2026/nime2026_98.pdf},
 year = {2026}
}