SwimTunes: A gamified music performance system for co-creating with a novice audience

Thomas Studley

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

This paper presents SwimTunes, a prototype game system designed for novice multi-user music-making in live performance settings. The system features a digital game and a public web app that allows audience members to participate using their mobile devices. After connecting via QR code, participants create and pilot virtual fish that generate music as they bump into one another. The performer then enters the game as a shark, using camera-based hand tracking to chase and consume the participants’ fish. The result is a performance dynamic that evolves from playful co-creation to one of gameful contest between the performer and audience. SwimTunes explores how this shifting interaction context can shape the instantiation of a set of musical parameters, and further how performers can harness gameplay metaphors to conduct live audiences in shared acts of musical expression. The paper details the design considerations and conceptual motivations that informed SwimTunes before describing its implementation via Node.js, Open Sound Control, Unreal Engine 5, and MetaSounds. It discusses technical challenges and opportunities unearthed during development and outlines future directions for the project and gamified music performance at large.

Citation

Thomas Studley. 2025. SwimTunes: A gamified music performance system for co-creating with a novice audience. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.15698912 [PDF]

BibTeX Entry

@article{nime2025_59,
 abstract = {This paper presents SwimTunes, a prototype game system designed for novice multi-user music-making in live performance settings. The system features a digital game and a public web app that allows audience members to participate using their mobile devices. After connecting via QR code, participants create and pilot virtual fish that generate music as they bump into one another. The performer then enters the game as a shark, using camera-based hand tracking to chase and consume the participants’ fish. The result is a performance dynamic that evolves from playful co-creation to one of gameful contest between the performer and audience. SwimTunes explores how this shifting interaction context can shape the instantiation of a set of musical parameters, and further how performers can harness gameplay metaphors to conduct live audiences in shared acts of musical expression. The paper details the design considerations and conceptual motivations that informed SwimTunes before describing its implementation via Node.js, Open Sound Control, Unreal Engine 5, and MetaSounds. It discusses technical challenges and opportunities unearthed during development and outlines future directions for the project and gamified music performance at large.},
 address = {Canberra, Australia},
 articleno = {59},
 author = {Thomas Studley},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.15698912},
 editor = {Doga Cavdir and Florent Berthaut},
 issn = {2220-4806},
 month = {June},
 numpages = {7},
 pages = {415--421},
 title = {SwimTunes: A gamified music performance system for co-creating with a novice audience},
 track = {Paper},
 url = {http://nime.org/proceedings/2025/nime2025_59.pdf},
 year = {2025}
}