Towards Note-Level Prediction for Networked Music Performance

Reid Oda, Adam Finkelstein, and Rebecca Fiebrink

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract:

The Internet allows musicians and other artists to collaborate remotely.However, network latency presents a fundamental challenge for remotecollaborators who need to coordinate and respond to each other's performancein real time. In this paper, we investigate the viability of predictingpercussion hits before they have occurred, so that information about thepredicted drum hit can be sent over a network, and the sound can be synthesizedat a receiver's location at approximately the same moment the hit occurs atthe sender's location. Such a system would allow two percussionists to playin perfect synchrony despite the delays caused by computer networks. Toinvestigate the feasibility of such an approach, we record vibraphone malletstrikes with a high-speed camera and track the mallet head position. We showthat 30 ms before the strike occurs, it is possible to predict strike time andvelocity with acceptable accuracy. Our method fits a second-order polynomial tothe data to produce a strike time prediction that is within the bounds ofperceptual synchrony, and a velocity estimate that will enable the soundpressure level of the synthesized strike to be accurate within 3 dB.

Citation:

Reid Oda, Adam Finkelstein, and Rebecca Fiebrink. 2013. Towards Note-Level Prediction for Networked Music Performance. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.1178624

BibTeX Entry:

  @inproceedings{Oda2013,
 abstract = {The Internet allows musicians and other artists to collaborate remotely.However, network latency presents a fundamental challenge for remotecollaborators who need to coordinate and respond to each other's performancein real time. In this paper, we investigate the viability of predictingpercussion hits before they have occurred, so that information about thepredicted drum hit can be sent over a network, and the sound can be synthesizedat a receiver's location at approximately the same moment the hit occurs atthe sender's location. Such a system would allow two percussionists to playin perfect synchrony despite the delays caused by computer networks. Toinvestigate the feasibility of such an approach, we record vibraphone malletstrikes with a high-speed camera and track the mallet head position. We showthat 30 ms before the strike occurs, it is possible to predict strike time andvelocity with acceptable accuracy. Our method fits a second-order polynomial tothe data to produce a strike time prediction that is within the bounds ofperceptual synchrony, and a velocity estimate that will enable the soundpressure level of the synthesized strike to be accurate within 3 dB.},
 address = {Daejeon, Republic of Korea},
 author = {Reid Oda and Adam Finkelstein and Rebecca Fiebrink},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.1178624},
 issn = {2220-4806},
 keywords = {Networked performance, prediction, computer vision},
 month = {May},
 pages = {94--97},
 publisher = {Graduate School of Culture Technology, KAIST},
 title = {Towards Note-Level Prediction for Networked Music Performance},
 url = {http://www.nime.org/proceedings/2013/nime2013_258.pdf},
 year = {2013}
}