P(l)aying Attention: Multi-modal, multi-temporal music control

Nicolas E Gold, Chongyang Wang, Temitayo Olugbade, Nadia Berthouze, and Amanda Williams

Proceedings of the International Conference on New Interfaces for Musical Expression

• Year: 2020
• Location: Birmingham, UK
• Pages: 172–175
• DOI: 10.5281/zenodo.4813303 (Link to paper)
• PDF link

Abstract:

The expressive control of sound and music through body movements is well-studied. For some people, body movement is demanding, and although they would prefer to express themselves freely using gestural control, they are unable to use such interfaces without difficulty. In this paper, we present the P(l)aying Attention framework for manipulating recorded music to support these people, and to help the therapists that work with them. The aim is to facilitate body awareness, exploration, and expressivity by allowing the manipulation of a pre-recorded ‘ensemble’ through an interpretation of body movement, provided by a machine-learning system trained on physiotherapist assessments and movement data from people with chronic pain. The system considers the nature of a person’s movement (e.g. protective) and offers an interpretation in terms of the joint-groups that are playing a major role in the determination at that point in the movement, and to which attention should perhaps be given (or the opposite at the user’s discretion). Using music to convey the interpretation offers informational (through movement sonification) and creative (through manipulating the ensemble by movement) possibilities. The approach offers the opportunity to explore movement and music at multiple timescales and under varying musical aesthetics.

Citation:

BibTeX Entry:

  @inproceedings{NIME20_33,
 abstract = {The expressive control of sound and music through body movements is well-studied.  For some people, body movement is demanding, and although they would prefer to express themselves freely using gestural control, they are unable to use such interfaces without difficulty.  In this paper, we present the P(l)aying Attention framework for manipulating recorded music to support these people, and to help the therapists that work with them. The aim is to facilitate body awareness, exploration, and expressivity by allowing the manipulation of a pre-recorded ‘ensemble’ through an interpretation of body movement, provided by a machine-learning system trained on physiotherapist assessments and movement data from people with chronic pain.  The system considers the nature of a person’s movement (e.g. protective) and offers an interpretation in terms of the joint-groups that are playing a major role in the determination at that point in the movement, and to which attention should perhaps be given (or the opposite at the user’s discretion).  Using music to convey the interpretation offers informational (through movement sonification) and creative (through manipulating the ensemble by movement) possibilities.  The approach offers the opportunity to explore movement and music at multiple timescales and under varying musical aesthetics.},
 address = {Birmingham, UK},
 author = {Gold, Nicolas E and Wang, Chongyang and Olugbade, Temitayo and Berthouze, Nadia and Williams, Amanda},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.4813303},
 editor = {Romain Michon and Franziska Schroeder},
 issn = {2220-4806},
 month = {July},
 pages = {172--175},
 publisher = {Birmingham City University},
 title = {P(l)aying Attention: Multi-modal, multi-temporal music control},
 url = {https://www.nime.org/proceedings/2020/nime2020_paper33.pdf},
 year = {2020}
}