Spatial 3D audio technologies are helping headset-wearers locate sounds beyond the field of view.
Recognizing where a sound comes from — a gunshot, a car horn, a dog’s bark — is key to how we understand our environment, and respond to it.
And that’s true in virtual reality, just as in real life.
But sound cues, which are influenced by everything from head size to sound pressure differences from one ear to another, are incredibly difficult to replicate in a virtual environment.
That’s why audio has become an important area where VR leaders are focused on innovation. The latest advances in VR audio technology – including sophisticated “spatial audio” solutions – are making improved sound “localization” possible in virtual environments.
– How the brain ‘localizes’ sound position
– Capturing audio in three dimensions
– Recognizing how our heads move
– Encoding your head into your audio
,
,
CREDIT: Ossic
How the brain ‘localizes’ sound position
Our brains process sound and visual inputs in very different ways, posing challenges for simulating alternate reality in VR.
When we use our eyes to see, the receptor cells of our retinas recognize the way light illuminates virtual or physical objects. That information is then communicated to the brain, which uses it to recognize the locations of visuals in the path of sight.
So when it comes to visuals, we still “see” VR media through the same process we use to see anything else.
Audio is a different story.
Sound location, in particular, is computed by the brain based on cues for things like direction and distance. We recognize lateral direction – aka a sound’s position to our right or left – using factors such as:
– Interaural time difference (ITD), the difference between the times sounds reach the two ears
– Interaural level difference (ILD), the difference in sound pressure level reaching the two ears
And that’s just right to left. Our brains also need to recognize the location of sounds in front or back of us, or above or below us.
To do that “localization,” we clue into the ways that sounds originating from different directions interact with the geometry of our bodies differently: The unique shapes and sizes of our heads, necks, torsos, and outer ears cause changes in sound filters and reflections – known as spectral modifications – that our brains use to infer direction and elevation.
These localization cues are just a few of the factors our brains consider when determining sound position. Distance localization is another concern, which we compute using cues around loudness, reverberations, and the time delay between an event taking place and the sound reaching our ears.
All of these considerations (among others) play into the development of VR audio technologies, which use specialized recording systems and algorithms to mimic natural audio.
Capturing audio in three dimensions
Audio companies have long tried to recreate location cues to better emulate real-world audio. One of the most effective approaches, which has made its way into VR, is “binaural” or 3D audio.
Binaural literally means “two ears,” but binaural audio is a far cry from traditional audio.
Binaural recording systems attempt to recreate the way the human head captures sound: Audio is recorded through microphones placed on both sides of a stand or dummy head (on the spots where our ears are) – enabling the microphones to capture level and time differences (aka ITD and ILD) just like our actual ears would.
Binaural microphones are typically housed in cavities that look just like the outer ear, which further improves their ability to capture audio in a way that sounds natural upon playback.
(Our outer ears, or pinnae, dramatically alter incoming sound waves; our brains understand these alterations as directional cues.)
,
,
Binaural audio has been around for centuries, but the rise of headphones and handheld audio devices has brought it back to the fore.
Whereas earlier eras of audio were about filling up big spaces with sound, headphone-delivered audio (including in VR) can benefit from the sense of 3D immersion that binaural audio provides.
“Binaural audio is critical to an immersive experience within the context of VR. We consider audio to be 50 percent of the immersive experience.”
–Adam Somers, Engineering Manager of JauntVR
But VR isn’t the only application. On YouTube, there are hundreds of binaural audio clips that are used for relaxation, studying, entertainment, and other purposes.
The effects are remarkable: In the “virtual haircut” video below, for example, you can hear the sounds of trimming behind you or atop your head. (Wear headphones to hear it properly.)
,
Source: CBINSIGHTS