The Phone with the Flow: Combining Touch + Optical Flow in Mobile Instruments

Cagan Arslan, Florent Berthaut, Jean Martinet, Ioan Marius Bilasco, and Laurent Grisoni

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

Mobile devices have been a promising platform for musical performance thanks to the various sensors readily available on board. In particular, mobile cameras can provide rich input as they can capture a wide variety of user gestures or environment dynamics. However, this raw camera input only provides continuous parameters and requires expensive computation. In this paper, we propose to combine motion/gesture input with the touch input, in order to filter movement information both temporally and spatially, thus increasing expressiveness while reducing computation time. We present a design space which demonstrates the diversity of interactions that our technique enables. We also report the results of a user study in which we observe how musicians appropriate the interaction space with an example instrument.

Citation

Cagan Arslan, Florent Berthaut, Jean Martinet, Ioan Marius Bilasco, and Laurent Grisoni. 2018. The Phone with the Flow: Combining Touch + Optical Flow in Mobile Instruments. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.1302709

BibTeX Entry

@inproceedings{Arslan2018,
 abstract = {Mobile devices have been a promising platform for musical performance thanks to the various sensors readily available on board. In particular, mobile cameras can provide rich input as they can capture a wide variety of user gestures or environment dynamics. However, this raw camera input only provides continuous parameters and requires expensive computation. In this paper, we propose to combine motion/gesture input with the touch input, in order to filter movement information both temporally and spatially, thus increasing expressiveness while reducing computation time. We present a design space which demonstrates the diversity of interactions that our technique enables. We also report the results of a user study in which we observe how musicians appropriate the interaction space with an example instrument.},
 address = {Blacksburg, Virginia, USA},
 author = {Cagan Arslan and Florent Berthaut and Jean Martinet and Ioan Marius Bilasco and Laurent Grisoni},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.1302709},
 editor = {Luke Dahl, Douglas Bowman, Thomas Martin},
 isbn = {978-1-949373-99-8},
 issn = {2220-4806},
 month = {June},
 pages = {148--151},
 publisher = {Virginia Tech},
 title = {The Phone with the Flow: Combining Touch + Optical Flow in Mobile Instruments},
 url = {http://www.nime.org/proceedings/2018/nime2018_paper0032.pdf},
 year = {2018}
}