Project links
Overview
"Room For Sound?" is a musical VR experience developed for Advanced Graphics and Interaction course at KTH. Through this project, we aim to learn and explore VR interaction with sound. The project has been displayed at ForskarFredag and Tekniska Museet in Stockholm. The project was developed by 5 people.
The player can choose to play between various percussive instruments, such as drums, maracas, and xylophone. There are backing tracks as well if the player wishes to play with them. On top of that, audiences can join the game as in-game avatars and can send messages and perform animations such as dancing and cheering through emotes.
Development
During the development process, I am responsible in developing the interface for the audience to join the game. The interface is a website that I developed using React, and the communication between the website and the game is established using HTTP calls, which I developed using NodeJS and ExpressJS.
I made APIs that allow the audiences to enter their names and send messages to the game. The messages they sent will then be displayed onto the VR environment which the player can see. The emotes, on the other hand, are predefined messages which will be processed first. The emote messages will trigger animations for their respective avatars, as seen in the image above.
A recurring problem is that the audience can only see their avatars when the VR player is looking at them, since initially we are streaming the VR's perspective to a large screen. When the VR player is looking away from the audiences, the audience cannot see their avatars at all.
To combat this problem, we implemented a new, static camera, looking at the audience dance floor. The static camera is then streamed to the audience website, so the audience can see their avatars from their phones. The static camera is streamed using Unity's Render Streaming package.