Reactive Video: Movement Sonification for Learning Physical Activity with Adaptive Video Playback

Doga Cavdir, Chris Clarke, Patrick Chiu, Laurent Denoue, and Don Kimber

Proceedings of the International Conference on New Interfaces for Musical Expression

• Year: 2021
• Location: Shanghai, China
• Article Number: 14
• DOI: 10.21428/92fbeb44.eef53755 (Link to paper)
• PDF link
• Presentation Video

Abstract:

This paper provides initial efforts in developing and evaluating a real-time movement sonification framework for physical activity practice and learning. Reactive Video provides an interactive, vision-based, adaptive video playback with auditory feedback on users' performance to better support when learning and practicing new physical skills. We implement the sonification for auditory feedback design by extending the Web Audio API framework. The current application focuses on Tai-Chi performance and provides two main audio cues to users for several Tai Chi exercises. We provide our design approach, implementation, and sound generation and mapping, specifically for interactive systems with direct video manipulation. Our observations reveal the relationship between the movement-to-sound mapping and characteristics of the physical activity.

Citation:

BibTeX Entry:

  @inproceedings{NIME21_14,
 abstract = {This paper provides initial efforts in developing and evaluating a real-time movement sonification framework for physical activity practice and learning. Reactive Video provides an interactive, vision-based, adaptive video playback with auditory feedback on users' performance to better support when learning and practicing new physical skills. We implement the sonification for auditory feedback design by extending the Web Audio API framework. The current application focuses on Tai-Chi performance and provides two main audio cues to users for several Tai Chi exercises. We provide our design approach, implementation, and sound generation and mapping, specifically for interactive systems with direct video manipulation. Our observations reveal the relationship between the movement-to-sound mapping and characteristics of the physical activity.},
 address = {Shanghai, China},
 articleno = {14},
 author = {Cavdir, Doga and Clarke, Chris and Chiu, Patrick and Denoue, Laurent and Kimber, Don},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.21428/92fbeb44.eef53755},
 issn = {2220-4806},
 month = {June},
 presentation-video = {https://youtu.be/pbvZI80XgEU},
 title = {Reactive Video: Movement Sonification for Learning Physical Activity with Adaptive Video Playback},
 url = {https://nime.pubpub.org/pub/dzlsifz6},
 year = {2021}
}