Skip to content

A Turing Test for B-Keeper : Evaluating an Interactive

Andrew Robertson, Mark D. Plumbley, and Nick Bryan-Kinns

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

Off-line beat trackers are often compared to human tappers who provide a ground truth against which they can be judged. In order to evaluate a real-time beat tracker, we have taken the paradigm of the 'Turing Test' in which an interrogator is asked to distinguish between human and machine. A drummer plays in succession with an interactive accompaniment that has one of three possible tempocontrollers (the beat tracker, a human tapper and a steadytempo metronome). The test is double-blind since the researchers do not know which controller is currently functioning. All participants are asked to rate the accompaniment and to judge which controller they believe was responsible. This method for evaluation enables the controllers to be contrasted in a more quantifiable way than the subjective testimony we have used in the past to evaluate the system. The results of the experiment suggest that the beat tracker and a human tapper are both distinguishable from a steadytempo accompaniment and they are preferable according to the ratings given by the participants. Also, the beat tracker and a human tapper are not sufficiently distinguishable by any of the participants in the experiment, which suggests that the system is comparable in performance to a human tapper.

Citation

Andrew Robertson, Mark D. Plumbley, and Nick Bryan-Kinns. 2008. A Turing Test for B-Keeper : Evaluating an Interactive. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.1179619 [PDF]

BibTeX Entry


@inproceedings{nime2008_Robertson,
author = {Robertson, Andrew and Plumbley, Mark D. and Bryan-Kinns, Nick},
title = {A Turing Test for B-Keeper : Evaluating an Interactive},
pages = {319--324},
booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
year = {2008},
address = {Genoa, Italy},
issn = {2220-4806},
doi = {10.5281/zenodo.1179619},
url = {https://nime.org/proceedings/2008/nime2008_319.pdf},
keywords = {Automatic Accompaniment, Beat Tracking, Human-Computer Interaction, Musical Interface Evaluation },
abstract = {Off-line beat trackers are often compared to human tappers who provide a ground truth against which they can be judged. In order to evaluate a real-time beat tracker, we have taken the paradigm of the 'Turing Test' in which an interrogator is asked to distinguish between human and machine. A drummer plays in succession with an interactive accompaniment that has one of three possible tempocontrollers (the beat tracker, a human tapper and a steadytempo metronome). The test is double-blind since the researchers do not know which controller is currently functioning. All participants are asked to rate the accompaniment and to judge which controller they believe was responsible. This method for evaluation enables the controllers to be contrasted in a more quantifiable way than the subjective testimony we have used in the past to evaluate the system. The results of the experiment suggest that the beat tracker and a human tapper are both distinguishable from a steadytempo accompaniment and they are preferable according to the ratings given by the participants. Also, the beat tracker and a human tapper are not sufficiently distinguishable by any of the participants in the experiment, which suggests that the system is comparable in performance to a human tapper.}
}