Joakinator: An Interface for Transforming Body Movement and Perception through Machine Learning and Sonification of Muscle-Tone and Force.

Joaquín R. Díaz Durán, Laia Turmo Vidal, and Ana Tajadura-Jiménez

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract:

Joakinator is a wearable interactive interface that allows users to activate different media materials, such as sound, music, and video, through body gestures. The device, designed in the context of music and performing arts, integrates surface electromyogram, force sensors, and machine learning algorithms with tailored-made software for sonifying muscle-tone and force. This allows the body to reflect expressively the content of the media and the architecture of the device. Recently, we have started to investigate the potential of Joakinator to alter body perception in the context of Joakinator, a European Research Council Project focused on the transformations of body perception through the use of interactive sound/haptics technology. At NIME-2023, we will showcase Joakinator and invite visitors to experience the device firsthand. Visitors will have the opportunity to try on the device, observe others using it, and reflect on its capabilities to transform body movement and perception through the sonification of muscle-tone and force. Overall, Joakinator is a technology that pushes the boundaries of body-computer interaction and opens new possibilities for human-computer interaction and expression.

Citation:

Joaquín R. Díaz Durán, Laia Turmo Vidal, and Ana Tajadura-Jiménez. 2023. Joakinator: An Interface for Transforming Body Movement and Perception through Machine Learning and Sonification of Muscle-Tone and Force.. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.11189114

BibTeX Entry:

  @inproceedings{nime2023_12,
 abstract = {Joakinator is a wearable interactive interface that allows users to activate different media materials, such as sound, music, and video, through body gestures. The device, designed in the context of music and performing arts, integrates surface electromyogram, force sensors, and machine learning algorithms with tailored-made software for sonifying muscle-tone and force. This allows the body to reflect expressively the content of the media and the architecture of the device. Recently, we have started to investigate the potential of Joakinator to alter body perception in the context of Joakinator, a European Research Council Project focused on the transformations of body perception through the use of interactive sound/haptics technology. At NIME-2023, we will showcase Joakinator and invite visitors to experience the device firsthand. Visitors will have the opportunity to try on the device, observe others using it, and reflect on its capabilities to transform body movement and perception through the sonification of muscle-tone and force. Overall, Joakinator is a technology that pushes the boundaries of body-computer interaction and opens new possibilities for human-computer interaction and expression.},
 address = {Mexico City, Mexico},
 articleno = {12},
 author = {Joaquín R. Díaz Durán and Laia Turmo Vidal and Ana Tajadura-Jiménez},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.11189114},
 editor = {Miguel Ortiz and Adnan Marquez-Borbon},
 issn = {2220-4806},
 month = {May},
 numpages = {4},
 pages = {94--97},
 title = {Joakinator: An Interface for Transforming Body Movement and Perception through Machine Learning and Sonification of Muscle-Tone and Force.},
 track = {Demos},
 url = {http://nime.org/proceedings/2023/nime2023_12.pdf},
 year = {2023}
}