CAVI: A Coadaptive Audiovisual Instrument–Composition

Cagri Erdem, Benedikte Wallace, and Alexander Refsum Jensenius

Proceedings of the International Conference on New Interfaces for Musical Expression

• Year: 2022
• Location: The University of Auckland, New Zealand
• Article Number: 25
• DOI: 10.21428/92fbeb44.803c24dd (Link to paper)
• PDF link
• Presentation Video

Abstract:

This paper describes the development of CAVI, a coadaptive audiovisual instrument for collaborative humanmachine improvisation. We created this agent-based live processing system to explore how a machine can interact musically based on a human performer’s bodily actions. CAVI utilized a generative deep learning model that monitored muscle and motion data streamed from a Myo armband worn on the performer’s forearm. The generated control signals automated layered time-based effects modules and animated a virtual body representing the artificial agent. In the final performance, two expert musicians (a guitarist and a drummer) performed with CAVI. We discuss the outcome of our artistic exploration, present the scientific methods it was based on, and reflect on developing an interactive system that is as much an audiovisual composition as an interactive musical instrument.

Citation:

BibTeX Entry:

  @inproceedings{NIME22_25,
 abstract = {This paper describes the development of CAVI, a coadaptive audiovisual instrument for collaborative humanmachine improvisation. We created this agent-based live processing system to explore how a machine can interact musically based on a human performer’s bodily actions. CAVI utilized a generative deep learning model that monitored muscle and motion data streamed from a Myo armband worn on the performer’s forearm. The generated control signals automated layered time-based effects modules and animated a virtual body representing the artificial agent. In the final performance, two expert musicians (a guitarist and a drummer) performed with CAVI. We discuss the outcome of our artistic exploration, present the scientific methods it was based on, and reflect on developing an interactive system that is as much an audiovisual composition as an interactive musical instrument.},
 address = {The University of Auckland, New Zealand},
 articleno = {25},
 author = {Erdem, Cagri and Wallace, Benedikte and Refsum Jensenius, Alexander},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.21428/92fbeb44.803c24dd},
 issn = {2220-4806},
 month = {jun},
 pdf = {176.pdf},
 presentation-video = {https://youtu.be/WO766vmghcQ},
 title = {{CAVI}: A Coadaptive Audiovisual Instrument{\textendash}Composition},
 url = {https://doi.org/10.21428%2F92fbeb44.803c24dd},
 year = {2022}
}