A web-based 3D environment for gestural interaction with virtual music instruments as a STEAM education tool

Kosmas Kritsis, Aggelos Gkiokas, Carlos Árpád Acosta, Quentin Lamerand, Robert Piéchaud, Maximos Kaliakatsos-Papakostas, and Vassilis Katsouros

Proceedings of the International Conference on New Interfaces for Musical Expression

Abstract:

We present our work in progress on the development of a web-based system for music performance with virtual instruments in a virtual 3D environment, which provides three means of interaction (i.e physical, gestural and mixed), using tracking data from a Leap Motion sensor. Moreover, our system is integrated as a creative tool within the context of a STEAM education platform that promotes science learning through musical activities. The presented system models string and percussion instruments, with realistic sonic feedback based on Modalys, a physical model-based sound synthesis engine. Our proposal meets the performance requirements of real-time interactive systems and is implemented strictly with web technologies.

Citation:

Kosmas Kritsis, Aggelos Gkiokas, Carlos Árpád Acosta, Quentin Lamerand, Robert Piéchaud, Maximos Kaliakatsos-Papakostas, and Vassilis Katsouros. 2018. A web-based 3D environment for gestural interaction with virtual music instruments as a STEAM education tool. Proceedings of the International Conference on New Interfaces for Musical Expression. DOI: 10.5281/zenodo.1302613

BibTeX Entry:

  @inproceedings{Kritsis2018,
 abstract = {We present our work in progress on the development of a web-based system for music performance with virtual instruments in a virtual 3D environment, which provides three means of interaction (i.e physical, gestural and mixed), using tracking data from a Leap Motion sensor. Moreover, our system is integrated as a creative tool within the context of a STEAM education platform that promotes science learning through musical activities. The presented system models string and percussion instruments, with realistic sonic feedback based on Modalys, a physical model-based sound synthesis engine. Our proposal meets the performance requirements of real-time interactive systems and is implemented strictly with web technologies.},
 address = {Blacksburg, Virginia, USA},
 author = {Kosmas Kritsis and Aggelos Gkiokas and Carlos Árpád Acosta and Quentin Lamerand and Robert Piéchaud and Maximos Kaliakatsos-Papakostas and Vassilis Katsouros},
 booktitle = {Proceedings of the International Conference on New Interfaces for Musical Expression},
 doi = {10.5281/zenodo.1302613},
 editor = {Luke Dahl, Douglas Bowman, Thomas Martin},
 isbn = {978-1-949373-99-8},
 issn = {2220-4806},
 month = {June},
 pages = {348--349},
 publisher = {Virginia Tech},
 title = {A web-based 3D environment for gestural interaction with virtual music instruments as a STEAM education tool},
 url = {http://www.nime.org/proceedings/2018/nime2018_paper0075.pdf},
 year = {2018}
}