Sounds from Mismatch: Sensorimotor Prediction Error as Sonic Material in Augmented Reality

Domenico Stefani, Alberto Boem, and Luca Turchet

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

Sounds from Mismatch (SfM) is an augmented reality (AR) sound experience in which sound emerges from the mismatch between a user's bodily actions and the system's expectations of those actions. SfM builds an internal model of user interactions by maintaining individual predictors for future positions of multiple tracked hand joints. Rather than treating prediction error solely as a quantity to be minimized, SfM frames sensorimotor and expectation mismatch as an expressive material for interaction. Using hand tracking in AR, the system generates sound from the dynamic discrepancy between the user's movements and a continuously updated “ghost” representation of anticipated gestures. This mismatch is mapped to granular sound textures, producing an embodied auditory experience that unfolds through exploration rather than control. SfM invites users to attend to the shifting relationship between visual perception, proprioception, and action, foregrounding anticipation and deviation as core elements of musical interaction. This paper describes the conceptual framing, interaction design, and technical implementation of SfM, and discusses its implications for designing embodied AR sound experiences based on expectation rather than performative accuracy.

Citation

Domenico Stefani, Alberto Boem, and Luca Turchet. 2026. Sounds from Mismatch: Sensorimotor Prediction Error as Sonic Material in Augmented Reality. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.20784184 [PDF]

BibTeX Entry

@inproceedings{nime2026_57,
 abstract = {Sounds from Mismatch (SfM) is an augmented reality (AR) sound experience in which sound emerges from the mismatch between a user's bodily actions and the system's expectations of those actions. SfM builds an internal model of user interactions by maintaining individual predictors for future positions of multiple tracked hand joints. Rather than treating prediction error solely as a quantity to be minimized, SfM frames sensorimotor and expectation mismatch as an expressive material for interaction. Using hand tracking in AR, the system generates sound from the dynamic discrepancy between the user's movements and a continuously updated “ghost” representation of anticipated gestures. This mismatch is mapped to granular sound textures, producing an embodied auditory experience that unfolds through exploration rather than control. SfM invites users to attend to the shifting relationship between visual perception, proprioception, and action, foregrounding anticipation and deviation as core elements of musical interaction. This paper describes the conceptual framing, interaction design, and technical implementation of SfM, and discusses its implications for designing embodied AR sound experiences based on expectation rather than performative accuracy.},
 address = {London, United Kingdom},
 articleno = {57},
 author = {Domenico Stefani and Alberto Boem and Luca Turchet},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.20784184},
 editor = {Benedict Gaster and João Tragtenberg and Anna Xambó and Tom Mitchell},
 issn = {2220-4806},
 month = {June},
 note = {},
 numpages = {7},
 pages = {485--491},
 title = {Sounds from Mismatch: Sensorimotor Prediction Error as Sonic Material in Augmented Reality},
 track = {Paper},
 url = {http://nime.org/proceedings/2026/nime2026_57.pdf},
 year = {2026}
}