Gulf: Piano and Gesture-Controlled Live Electronics

Zubin Kanga, Rylan Gleave, and Tom Mitchell

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

This performance by Dr Zubin Kanga (Royal Holloway, University of London) features a new work for solo piano and electronics that demonstrates an innovative use of MiMU's gesture-control technologies, where the pianist's hand gestures control live electronics using webcam-based video-motion-capture rather than a sensor glove. Gulf (2025), composed by Rylan Gleave, utilises research undertaken by Dr Thomas J. Mitchell (UWE, Bristol) that links the Glover mapping software (created by MiMU for use with their sensor gloves and co-developed by Mitchell) with Google's MediaPipe (an open-source framework for creating real-time machine learning solutions for video-based motion and gesture tracking) to combine the flexibility and accessibility of single-RGB-camera motion capture with the end-user mapping capabilities of Glover. The result is a system which uses a camera, placed on the piano facing the pianist, to facilitate motion capture, allowing the performer to lift their hands off the keys to control electronic sounds through gestures. The combination of software allows the live piano sound to be shaped and parameters to be manipulated through movements of the hands across the range of camera vision. Multiple layers of control are facilitated by the use of trigger gestures and complex combinations of different types of gesture and movement. This work builds on previous research with motion-based instruments, including Tanaka (2009), Miranda and Wanderley (2006) and Frid (2019), as well as research into MediaPipe-based gesture mapping for music, including Lim, Kotsani and Hartono (2022), Cochiara and Turchet (2022), and Bayd et al. (2022). It also builds on the skills and music-technology explorations developed in works composed by, or created in collaboration with, Dr Zubin Kanga, integrating gesture into performance using sensors and new interfaces (including the MiMU gloves), such as Neil Luck and Zubin Kanga’s Whatever Weighs You Down (2022), Jon Rose’s Ballast (2019), and Laura Bowler’s SHOW(ti)ME (2022). The performance showcases a unique combination of technologies and the integration of instrumental performance with camera-based gestural control, while also demonstrating the level of technique and virtuosity required to explore the full expressive potential of the technologies in performance.

Citation

Zubin Kanga, Rylan Gleave, and Tom Mitchell. 2026. Gulf: Piano and Gesture-Controlled Live Electronics. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.20782166 [PDF]

BibTeX Entry

@inproceedings{nime2026_music_45,
 abstract = {This performance by Dr Zubin Kanga (Royal Holloway, University of London) features a new work for solo piano and electronics that demonstrates an innovative use of MiMU's gesture-control technologies, where the pianist's hand gestures control live electronics using webcam-based video-motion-capture rather than a sensor glove. Gulf (2025), composed by Rylan Gleave, utilises research undertaken by Dr Thomas J. Mitchell (UWE, Bristol) that links the Glover mapping software (created by MiMU for use with their sensor gloves and co-developed by Mitchell) with Google's MediaPipe (an open-source framework for creating real-time machine learning solutions for video-based motion and gesture tracking) to combine the flexibility and accessibility of single-RGB-camera motion capture with the end-user mapping capabilities of Glover. The result is a system which uses a camera, placed on the piano facing the pianist, to facilitate motion capture, allowing the performer to lift their hands off the keys to control electronic sounds through gestures. The combination of software allows the live piano sound to be shaped and parameters to be manipulated through movements of the hands across the range of camera vision. Multiple layers of control are facilitated by the use of trigger gestures and complex combinations of different types of gesture and movement. This work builds on previous research with motion-based instruments, including Tanaka (2009), Miranda and Wanderley (2006) and Frid (2019), as well as research into MediaPipe-based gesture mapping for music, including Lim, Kotsani and Hartono (2022), Cochiara and Turchet (2022), and Bayd et al. (2022). It also builds on the skills and music-technology explorations developed in works composed by, or created in collaboration with, Dr Zubin Kanga, integrating gesture into performance using sensors and new interfaces (including the MiMU gloves), such as Neil Luck and Zubin Kanga’s Whatever Weighs You Down (2022), Jon Rose’s Ballast (2019), and Laura Bowler’s SHOW(ti)ME (2022). The performance showcases a unique combination of technologies and the integration of instrumental performance with camera-based gestural control, while also demonstrating the level of technique and virtuosity required to explore the full expressive potential of the technologies in performance.},
 address = {London, United Kingdom},
 articleno = {45},
 author = {Zubin Kanga and Rylan Gleave and Tom Mitchell},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.20782166},
 editor = {Lia Mice and Nicole Robson and Tara Pattenden},
 issn = {2220-4806},
 month = {June},
 note = {Live Performance},
 numpages = {3},
 pages = {192--194},
 presentation-video = {https://www.youtube.com/watch?v=vhQa5uy12XE},
 title = {Gulf: Piano and Gesture-Controlled Live Electronics},
 track = {Music},
 url = {http://nime.org/proceedings/2026/nime2026_music_45.pdf},
 year = {2026}
}