HarmonEyes: A 3D Soundscape Explored by Ear

Evan Strasnick, Jeffrey Snyder

Princeton University, 2015.

Typically, when entering a new space, users are able to quickly familiarize themselves with their surroundings via sight, or even touch. However, it is less common to understand an environment via the auditory sense. We created a system which maps the 3D volume of a room to a virtual soundscape for users to explore by ear. A Kinect sensor tracks the user's hand and sends its 3D coordinates to a Max/MSP program, modulating various parameters of the sound and rendering sound samples scattered in localized regions throughout the space. We then evaluated the system in an installation where users were invited to explore the space blindfolded. Users describe the experience of understanding their surroundings via sound, and then reconciling their mental map with post-hoc visual information.