Jump to content

Sound Mixing for VR/Oculus Film Making


Jesse Flaitz
 Share

Recommended Posts

I worked the TED conference in Vancouver a month ago and the Wired conference in NYC the other day and at both places I had some free time and was able to play around with the VR1 (similar to the Oculus Rift).  The first instance involved a completely CG video and, using headphones, I was a little disappointed in myself in retrospect.  I was so captivated by the video and the world that I failed to analyse the way they had used sound in the environment.  Unfortunately the second time I used the headset, while still impressive visually, there was no headphones available so I wasn't able to put the sound design to the test.

 

I started thinking about how sound would work in a completely free roaming 360 degree environment.  In video games for instance, objects have a programmed location and with headphones on, they can place that sound anywhere sonically that they want to.  But does it work that way with VR video?  When using VR headsets for video games, I'd assume that it works the same way it would on a normal screen, but for a narrative video, do the deliverables for VR formats lend themselves to the same level of manipulation?

 

If a film is shot entirely with 360 degree cameras, the information about what objects are where isn't programmed into the video.  For example, if there was an idling car in a scene, how would I place that sound so no matter where I was looking, that car would sound like it was in the same place while factoring in the inter-aural arrival time differences.

 

Is there currently software available that has the capability to program all objects to point within an environment for purpose of VR film making?

 

Hopefully my question is clear, it's a bit difficult to articulate given how little info I actually have about VR and video game sound in general.

Link to comment
Share on other sites

Are there commercially viable headphones that do surround? 

 

In terms of a film, I think it would be very hard. Live action would be near impossible to do this way. You'd have to shoot with 360 degree cameras and that means crew couldn't be anywhere and LOTS of post work to make it right. You could get away with this in animation. The sound would have to be approached like it is in video games. I've worked in games, so I know about how it all works. Each audio asset created would be a mono sound and then you'd have to place that in the 3D virtual environment using game implementation software. However, I'm so used to watching a movie or tv show via a flat screen I can't imagine watching something via VR. It would be extremely difficult because the director needs to control what we see, and if something important is happening off to your left, and you are looking right, you'll miss it. 

Link to comment
Share on other sites

For narrative, VR I'm sure would be very challenging, but I can imagine a new Planet Earth series being shot for VR and in that type of situation sound design would play a hugely important role.  I'm curious as to what software would be used for an application like that.

Link to comment
Share on other sites

For narrative, VR I'm sure would be very challenging, but I can imagine a new Planet Earth series being shot for VR and in that type of situation sound design would play a hugely important role.  I'm curious as to what software would be used for an application like that.

You'd have to approach it like a video game. Audio assets would be generated in a DAW and then implemented in a game engine. I don't know if this would work for a long form narrative, though. Custom development tools would need to be created for this sort of thing, and it would take a long time to develop them and then the workflow properly. 

Link to comment
Share on other sites

There are already projects in the works that will be shot "spherical", with the viewer able to orient ("face") themselves in the movie's space.   How the sound will be handled is a matter of layers of proprietary software that will change the sound field to match the viewer's virtual orientation within the sphere, plus the results of what the filmmaker will be trying to get the viewer to experience at that moment.  Editorial, sound mixing and the storytelling itself will be pretty different from what we know now.

 

p

Link to comment
Share on other sites

This is an excellent question and probably a bit more complicated solution might be required for shooting into the future with such apparatus. In my mind I would think the work flow might include some sort of movable grid of actual mics going to tracks, or a very nice piece of software that can model a space based on input from programmers concerning the materials the space is comprised of in combination with some sort of perspective mapping using LIDAR models or something similar. The Oculus stuff I've been able to put my hands on is not only 360 degrees, it is full virtual field of image wherever you happen to point your face and they are rapidly innovating ways to move yourself around in this virtual space, soon you will even be able to interact with your own hands that are tracked by the hardware. There is a wide open untouched area for new sound techniques coming in the future for stuff like this. It might be a fad, but then again who knows what the future holds.

Link to comment
Share on other sites

It's t he new 3D Todd;~)  Who knows really. It's cool technology no doubt but it seems like a gamers world or on a higher level a virtual amusement park in different worlds. I think our limited parts in this technology will be capturing the words of the onscreen actors. Same as it ever was...

CrewC

Link to comment
Share on other sites

Agreed Todd. That is if the sets are 3D record worthy (like having a unique sound or ambience) but most sets and filming processes are not that and sound design is at such a high state of the art it would be hard to beat it w real. IMO words from actors is our main job now and into the future. Not sure I'll be there but I will be following the changes to traditional media as well as new media.

CrewC

Link to comment
Share on other sites

I sometimes work for a production company here in NYC named Total Cinema 360 that deals solely in VR / Oculus Rift / 360 Video.

 

The work they produce is quite special because it does in fact use sound to sell the VR through 3D, binaural and spatialization. The process to achieving it is one that they developed (and continue to develop) in-house with their sound director, sound mixer, acoustician, and my good friend Laura Cunningham.

 

We use typical binaural and ambisonics recording techniques to sell camera/1st person perspective, as well as typical production sound recording techniques for the obvious stuff. But the trick is all in post, which is where all the VR magic happens. I'm unable to speak details about that part, because obviously that is the secret sauce that makes them special, but at the end of the process, they are able to deliver VR experiences with interactive sound.

 

The technology for this niche is evolving incredibly fast, and we're excited to see what the future holds.

Link to comment
Share on other sites

What would be cool is real time VR of a 360 degree camera in a remote location, and then robotic control of devices. I'm thinking of medical procedures, where you could have medical "robots" with the cameras in places in the world where it is hard to get surgeons to, and have the surgeon at their home base. A surgeon based on Seattle could treat a gunshot wound on someone in the Congo in the morning and then remove a tumor from someone in Siberia in the afternoon. 

Link to comment
Share on other sites

I have done a number of these VR shoots. Some production companies use a tetrahedral microphone, which is four hypercardioid mics arranged like a pyramid. The sounds that are recorded by these mics are, basically, four tracks with no manipulation. The process of encoding those four tracks is done in post.

 

The rest of the sounds (dialogue, Foley, etc.) is all recorded in mono, and sound designed after the fact.

 

HTH,

BK

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...