Jesse Flaitz Posted May 14, 2015 Report Share Posted May 14, 2015 I worked the TED conference in Vancouver a month ago and the Wired conference in NYC the other day and at both places I had some free time and was able to play around with the VR1 (similar to the Oculus Rift). The first instance involved a completely CG video and, using headphones, I was a little disappointed in myself in retrospect. I was so captivated by the video and the world that I failed to analyse the way they had used sound in the environment. Unfortunately the second time I used the headset, while still impressive visually, there was no headphones available so I wasn't able to put the sound design to the test. I started thinking about how sound would work in a completely free roaming 360 degree environment. In video games for instance, objects have a programmed location and with headphones on, they can place that sound anywhere sonically that they want to. But does it work that way with VR video? When using VR headsets for video games, I'd assume that it works the same way it would on a normal screen, but for a narrative video, do the deliverables for VR formats lend themselves to the same level of manipulation? If a film is shot entirely with 360 degree cameras, the information about what objects are where isn't programmed into the video. For example, if there was an idling car in a scene, how would I place that sound so no matter where I was looking, that car would sound like it was in the same place while factoring in the inter-aural arrival time differences. Is there currently software available that has the capability to program all objects to point within an environment for purpose of VR film making? Hopefully my question is clear, it's a bit difficult to articulate given how little info I actually have about VR and video game sound in general. Quote Link to comment Share on other sites More sharing options...
dfisk Posted May 14, 2015 Report Share Posted May 14, 2015 Are there commercially viable headphones that do surround? In terms of a film, I think it would be very hard. Live action would be near impossible to do this way. You'd have to shoot with 360 degree cameras and that means crew couldn't be anywhere and LOTS of post work to make it right. You could get away with this in animation. The sound would have to be approached like it is in video games. I've worked in games, so I know about how it all works. Each audio asset created would be a mono sound and then you'd have to place that in the 3D virtual environment using game implementation software. However, I'm so used to watching a movie or tv show via a flat screen I can't imagine watching something via VR. It would be extremely difficult because the director needs to control what we see, and if something important is happening off to your left, and you are looking right, you'll miss it. Quote Link to comment Share on other sites More sharing options...
Jesse Flaitz Posted May 14, 2015 Author Report Share Posted May 14, 2015 For narrative, VR I'm sure would be very challenging, but I can imagine a new Planet Earth series being shot for VR and in that type of situation sound design would play a hugely important role. I'm curious as to what software would be used for an application like that. Quote Link to comment Share on other sites More sharing options...
Chris Woodcock Posted May 14, 2015 Report Share Posted May 14, 2015 Are there commercially viable headphones that do surround? I have a 7.1 gaming headset, there are a few options. Quote Link to comment Share on other sites More sharing options...
dfisk Posted May 14, 2015 Report Share Posted May 14, 2015 For narrative, VR I'm sure would be very challenging, but I can imagine a new Planet Earth series being shot for VR and in that type of situation sound design would play a hugely important role. I'm curious as to what software would be used for an application like that. You'd have to approach it like a video game. Audio assets would be generated in a DAW and then implemented in a game engine. I don't know if this would work for a long form narrative, though. Custom development tools would need to be created for this sort of thing, and it would take a long time to develop them and then the workflow properly. Quote Link to comment Share on other sites More sharing options...
Philip Perkins Posted May 14, 2015 Report Share Posted May 14, 2015 There are already projects in the works that will be shot "spherical", with the viewer able to orient ("face") themselves in the movie's space. How the sound will be handled is a matter of layers of proprietary software that will change the sound field to match the viewer's virtual orientation within the sphere, plus the results of what the filmmaker will be trying to get the viewer to experience at that moment. Editorial, sound mixing and the storytelling itself will be pretty different from what we know now. p Quote Link to comment Share on other sites More sharing options...
old school Posted May 14, 2015 Report Share Posted May 14, 2015 The Oculus projects I've worked on we just recorded the talent old school. (mono mix and iso's) Post placed them in the visual field and the ambience was all sound design. CrewC Quote Link to comment Share on other sites More sharing options...
Todd Weaver Posted May 15, 2015 Report Share Posted May 15, 2015 This is an excellent question and probably a bit more complicated solution might be required for shooting into the future with such apparatus. In my mind I would think the work flow might include some sort of movable grid of actual mics going to tracks, or a very nice piece of software that can model a space based on input from programmers concerning the materials the space is comprised of in combination with some sort of perspective mapping using LIDAR models or something similar. The Oculus stuff I've been able to put my hands on is not only 360 degrees, it is full virtual field of image wherever you happen to point your face and they are rapidly innovating ways to move yourself around in this virtual space, soon you will even be able to interact with your own hands that are tracked by the hardware. There is a wide open untouched area for new sound techniques coming in the future for stuff like this. It might be a fad, but then again who knows what the future holds. Quote Link to comment Share on other sites More sharing options...
old school Posted May 15, 2015 Report Share Posted May 15, 2015 It's t he new 3D Todd;~) Who knows really. It's cool technology no doubt but it seems like a gamers world or on a higher level a virtual amusement park in different worlds. I think our limited parts in this technology will be capturing the words of the onscreen actors. Same as it ever was... CrewC Quote Link to comment Share on other sites More sharing options...
Todd Weaver Posted May 15, 2015 Report Share Posted May 15, 2015 Could very well be. But it would be a much better virtual world If the sounds were also captured in a way that would make the experience more realistic. Quote Link to comment Share on other sites More sharing options...
old school Posted May 15, 2015 Report Share Posted May 15, 2015 Agreed Todd. That is if the sets are 3D record worthy (like having a unique sound or ambience) but most sets and filming processes are not that and sound design is at such a high state of the art it would be hard to beat it w real. IMO words from actors is our main job now and into the future. Not sure I'll be there but I will be following the changes to traditional media as well as new media. CrewC Quote Link to comment Share on other sites More sharing options...
Ze Frias Posted May 15, 2015 Report Share Posted May 15, 2015 I sometimes work for a production company here in NYC named Total Cinema 360 that deals solely in VR / Oculus Rift / 360 Video. The work they produce is quite special because it does in fact use sound to sell the VR through 3D, binaural and spatialization. The process to achieving it is one that they developed (and continue to develop) in-house with their sound director, sound mixer, acoustician, and my good friend Laura Cunningham. We use typical binaural and ambisonics recording techniques to sell camera/1st person perspective, as well as typical production sound recording techniques for the obvious stuff. But the trick is all in post, which is where all the VR magic happens. I'm unable to speak details about that part, because obviously that is the secret sauce that makes them special, but at the end of the process, they are able to deliver VR experiences with interactive sound. The technology for this niche is evolving incredibly fast, and we're excited to see what the future holds. Quote Link to comment Share on other sites More sharing options...
dfisk Posted May 15, 2015 Report Share Posted May 15, 2015 What would be cool is real time VR of a 360 degree camera in a remote location, and then robotic control of devices. I'm thinking of medical procedures, where you could have medical "robots" with the cameras in places in the world where it is hard to get surgeons to, and have the surgeon at their home base. A surgeon based on Seattle could treat a gunshot wound on someone in the Congo in the morning and then remove a tumor from someone in Siberia in the afternoon. Quote Link to comment Share on other sites More sharing options...
Blas Kisic Posted May 20, 2015 Report Share Posted May 20, 2015 I have done a number of these VR shoots. Some production companies use a tetrahedral microphone, which is four hypercardioid mics arranged like a pyramid. The sounds that are recorded by these mics are, basically, four tracks with no manipulation. The process of encoding those four tracks is done in post. The rest of the sounds (dialogue, Foley, etc.) is all recorded in mono, and sound designed after the fact. HTH, BK Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.