VR game for composing music spatially




10 weeks

Project Type

VR game prototyping


spatial audio, gestural interaction, Unity


Ideation, Prototyping


Aria Xiying Bao, Nix Liu Xin, Emmanuel Serrano, Yinghou Wang

Musicraft is a VR audio-visual gaming experience and platform where the user creates and remixes music and architecture at the same time. It explores the idea that architecture is frozen music and music is flowing architecture.

Concept sketch by Yinghou Wang

Game Mechanics

Stack and build freely with architectural building blocks.

Each block is a musical track. Layer and remix music while building architecture.

Step into the audio-visual space and experience it immersively.

Interaction Design


The project is built with Unity. On the software side, we are utilizing Meta’s Oculus Integration SDK to develop and deploy the VR experience. Specifically, we use Oculus Interaction SDK to enable vision-based hand-tracking and leverage its hand interaction infrastructure (hand physics, collision, gestures, custom hand poses, etc.). Unity Shader Graphs are used in combination with custom materials and textures to render the desired toon visuals resembling that of the game Monument Valley.

A major technical task for the project is the object snapping mechanism when modeling. A snapping mechanism is necessary because of the limited effects we add to the audio tracks based on the way objects are combined. Some of the technical considerations are a 3D grid system and preset snapping slhouettes custimized to each object.


An immersive spatial audio-visual experience where the audience is inside a morphing building synced to the remixing of the music in real time.

Vision images by Nix Liu Xin

Designed with ❤️ across realities

Davide Zhang © 2016 - 2023. All Rights Reserved.

Davide Zhang © 2016 - 2023. All Rights Reserved.

Davide Zhang © 2016 - 2023. All Rights Reserved.