A Computer-aided Multimodal Music Learning System with Curriculum: A Pilot Study

Daniel Chin, and Gus Xia

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract

We present an AI-empowered music tutor with a systematic curriculum design. The tutoring system fully utilizes the interactivity space in the auditory, visual, and haptic modalities, supporting seven haptic feedback modes and four visual feedback modes. The combinations of those modes form different cross-modal tasks of varying difficulties, allowing the curriculum to apply the “scaffolding then fading” educational technique to foster active learning and amortize cognitive load. We study the effect of multimodal instructions, guidance, and feedback using a qualitative pilot study with two subjects over ~11 hours of training with our tutoring system. The study reveals valuable insights about the music learning process and points towards new features and learning modes for the next prototype.

Citation

Daniel Chin, and Gus Xia. 2022. A Computer-aided Multimodal Music Learning System with Curriculum: A Pilot Study. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.21428/92fbeb44.c6910363

BibTeX Entry

@inproceedings{NIME22_39,
 abstract = {We present an AI-empowered music tutor with a systematic curriculum design. The tutoring system fully utilizes the interactivity space in the auditory, visual, and haptic modalities, supporting seven haptic feedback modes and four visual feedback modes. The combinations of those modes form different cross-modal tasks of varying difficulties, allowing the curriculum to apply the “scaffolding then fading” educational technique to foster active learning and amortize cognitive load. We study the effect of multimodal instructions, guidance, and feedback using a qualitative pilot study with two subjects over ~11 hours of training with our tutoring system. The study reveals valuable insights about the music learning process and points towards new features and learning modes for the next prototype.},
 address = {The University of Auckland, New Zealand},
 articleno = {39},
 author = {Chin, Daniel and Xia, Gus},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.21428/92fbeb44.c6910363},
 issn = {2220-4806},
 month = {jun},
 pdf = {39.pdf},
 presentation-video = {https://youtu.be/DifOKvH1ErQ},
 title = {A Computer-aided Multimodal Music Learning System with Curriculum: A Pilot Study},
 url = {https://doi.org/10.21428%2F92fbeb44.c6910363},
 year = {2022}
}