Jump to content

VR production (sound), new like old?


Recommended Posts

There seems to be quite a vogue for VR production right now.  Some aspects of the work I've seen I don't quite understand the appeal of vs the work involved to do it--like the ability to look away from the talent at what's behind the camera, the floor, the ceiling...?   But it's early days--the great creative cinematic minds of our time haven't taken this challenge on yet, that I know of.  Meanwhile....how to record for this medium?   Cameras that see everywhere (I guess), the need (I think) for omni-hearing Ambisonic mic techniques....it struck me that a successful sync sound VR production would need to control the shooting environment much MORE than we usually do these days, even on sound stages.   It reminded me of the earliest sync sound Hollywood productions, those made before directional mics or even movable mic booms--the stages had to be quiet, the actors had to speak up, the camera and lighting equipment had to be near-silent, sets solid and non-resonant (or squeaky), and the crew et al had to be quiet during shots.   Is this a ridiculous notion, considering how far away from the above our shooting situations are now?  Like a more likely scenario is the use of even more wireless lavs and ADR (since it seems like boom mics might be an issue a lot of the time)?   Idle thoughts, but there is a lot of VR production going on around here all of a sudden....

Link to comment
Share on other sites

Interestingly enough, I'm seeing almost the opposite , where a lot of vr productions don't care about controlling the environment or the space, and the answer always is "it's real".  Which I feel has always been a poor excuse, but when  I bring it up from a quality control stand point , I hear, well, they could see it if they looked that direction, so it's got to be there.  We're here to capture what's actually happening, etc..

VR can be quite post intensive at times, and no matter what there is always some or a lot of painting out going on, so I see a lot of opportunity for creative problem solving.  On the other hand I also get a lot of requests with completely unrealistic expectations.

Link to comment
Share on other sites

Oh I've heard the "It's Real" thing my whole career: it generally translates to "we're too lazy to do anything about BG noise etc".  I wondered if a serious filmmaker would take up the challenge of trying to capture a full performance with real actors/sets etc, not the sort of cheapie corporate style productions I've seen using this technology so far.   Think "Rope", or "Playhouse 90" etc..  I agree about the 360 nature of the recording needing to be made to match picture, but so far the sync audio I've heard in these projects has been terrible, uncontrolled.  But you are probably right in that the answer will be MORE post mojo, not more careful production.

Link to comment
Share on other sites

Coincidentally, Laura Cunningham and myself will be doing a chat with Peter Schneider about VR / 360 sound recording on the next Gotham Sound TV live stream, next Tuesday at 2pm EST!

http://livestream.com/gothamsound/soundfield-vr-recording?t=1465585662

We will hopefully cover a fair bit about the production and post-production process for VR / 360 videos, and we will probably be able to take some questions, too.

Link to comment
Share on other sites

VR / 360 plays best as a medium when the goal of the content is to place the user / consumer in the middle of the action. It has to be experiential, and ideally interactive. Otherwise, it's gimmicky and boring, IMO. There should ideally be something happening all around, because then the viewer can choose what he/she wants to interact with. This is why I don't think it would necessarily play well for narrative situations, and why you see it be really successful as a medium for unscripted (though controlled) situations and environments.

Spatialization of point-sources is truly the best way to create the 3D sound environment required to convince your brain of the experience. Think of it as a video game, you have to record each individual sound, and spatialize them in relation to the 1st person perspective, achieving the proper HRTF to create the realistic sound. That said, some 3D sound recording tools that can help richen the 3D soundscape exist, including my personal favorite, the ambisonic format.

As it stands today, very few consumer platforms allow for a full 3D sound experience. Oculus and other VR headsets allow for a full 3D sound experience via customized apps. But the two most popular and readily available ones are YouTube and Facebook 360, and they are still working their ways to developing better sound experiences through their videos. They're both pretty close though. Facebook just bought Two Big Ears' Spatial Workstation plug-in for Pro Tools, which is what my colleagues in post are using these days. I think it will help drive improvements in their platform as well as push competition. I'm excited.

Link to comment
Share on other sites

Like anything new and trendy, VR has its place, but its place is not everywhere!

I did a VR gig for the Indy 500.  There were a few installed camera rigs and one portable one on a stand.  Mostly I just captured a variety of ambient sounds that post could use if desired.  A few times I accompanied the portable rig and captured things around it with me and the boom just sort of blending into the teeming crowd -- after all, media crews are a normal part of the event.

At NAB this year, Sennheiser had a VR demo within which the sound followed the view.  That obviously takes a good bit of planning and a lot of careful post.

Link to comment
Share on other sites

Here are some examples of stuff I did that I think is made ideal for the medium:

(Unfortunately this one is only a fixed stereo image, as YouTube did not support any 3D audio yet)

Here's one that we did a lot of underwater recording off the coast of Panamá:
https://littlstar.com/videos/b51755c2

And here, I bet you've never experienced a rocket launch this close:
https://littlstar.com/videos/8601543d

Link to comment
Share on other sites

It'd be a great way to do travel logs. Take people places that many could not visit easily, like mountain climbing, scuba diving, surfing, etc. Inside out VR seems interesting that way. Outside in VR, (360 ring of cameras looking in) of say a boxing ring w fighters going at it and the viewer could look at from any pov would lend itself to scenarios and or scenes better that inside out could.   

CrewC

Link to comment
Share on other sites

I have the Two Big Ears app, and have been working with it some.  As you say, the only sounds that really seem to sing "I'm Over Here" are mono sfx.  Stereo didn't work well at all, Ambisonic stiff was kind of everywhere.  Oddly, given that it sees "everything", I found that the soundtrack had to stay a lot simpler than a normal movie soundtrack for the illusions to hold--those key directional sfx have to be really clear and not subsumed in an ambient (or even music) bed to play well.  I still think narrative filmmakers are not going to be able to resist the idea of putting you "in" the scene of a fiction piece, as difficult as it will be.  Who will be the Orson Welles of VR?

Link to comment
Share on other sites

This is something I've developed a keen interest in as well, and have teamed up with some great post-production colleagues up here in Toronto to establish effective workflows. Right now it's still a lot of experimentation, and as you noted there has not been a lot of (good) narrative work done. For that side of things, almost all of the magic is done in post from mono recordings and sound design. The only effective examples I've seen so far have been in the horror genre, since it lends itself well to immersing the viewer in a tense environment. Spatial audio becomes even more important than ever, to ensure that the viewer is looking in the right direction at the right time. It's actually pretty fun to play around with.

The short piece "Help" by Justin Lin and Google Spotlight is a pretty cool example too of how live-action blockbusters could be done in VR. I can't imagine anyone would want to watch something more than 5 or 10 minutes long though, at least until the wearable tech vastly improves.

Aside from narrative work, there's some fun to be had in recording "immersive experiences" for sports, music, or travel fans. Imagine being in the locker room before a big game. This adds a bit of interest for us to record too, since you can play with the mix between ambisonic recordings and spot (lav) recordings of key people like the coach addressing the players. Finding that sweet spot in the mix that puts the viewer in the middle of the excitement but also maintains clear, consistent dialogue is a tricky challenge, but a good one! Of course, that's all up to the post team, our job is still limited to recording the best sounds possible and delivering it to them. But more than ever before, communcation between the production sound mixer and post-production team is of paramount importance during pre-production, to agree on a workflow that works for each project.

Link to comment
Share on other sites

It seems that most of the "mixing" is done spatially in post (aka 3D audio).  Hell, Facebook even bought a company (they own Oculus but also bought an audio company that now has a free 3D audio Pro Tools HD plugin.  I recently did a short narrative VR shoot that had stunts and everything else you'd expect from an action film, but because of the environment (a big sailing boat that was at sea while shooting), we had to get creative due to the elements AND since you can't just boom things.  I lav'ed everyone up like usual, and then creatively hid wireless plant mics.  I also got a ton of wild lines once I got back on shore.  It should be premiering online on the 15th I'm told so I'm curious myself to see how it turned out since this was my first rodeo doing a VR shoot!

Link to comment
Share on other sites

Did you do 4-channel (or etc) ambiances, or will post figure that out?  I guess my question about trying to get 4 channel or Ambisonic production sound is that the set+location noise could be all over the recording--there is no way to pattern anything off as with a shotgun boom mic, so maybe it wouldn't be worth bothering with (esp since all talent will have to be wired plus the plants etc, so lots to do already).

Link to comment
Share on other sites

35 minutes ago, Philip Perkins said:

Did you do 4-channel (or etc) ambiances, or will post figure that out?

We didn't do any ambiances / ambisonic recordings.  Post told me that they were going to "spatialize" all the voices / effects and that the most important thing to get was just clean dialogue on isolated tracks for each character so that the software they use could literally say "character 1 is currently 5 feet from camera" and then as he walks to ten feet left of camera, they can automate that so that it sounds as though the character really is moving across the frame (the software can place a character / effect in an X,Y, and Z plane kind of like Dolby's Atmos).  I haven't heard/seen any of it yet but I'm really curious as to how well that actually sounds because it definitely sounds interesting!

Link to comment
Share on other sites

Thanks to Gotham, Laura, and Jose for doing the VR talk, I look forward to watching it.

One thing I do think will be important to filmmakers is the use of spatial audio to help guide the viewer where to look.  If an important moment in a scene is going to happen to one side, or behind you, it'd be very easy to miss, but if the sound for that moment starts early, a new sound in a new direction will catch your attention and likely cause you to look.

Here's a video that I recorded a little while ago, that I think is a fun use for VR.  A VR cam is a good way to put people in places or events where they wouldn't norrmally be able to be.  I used a Binaural head that mostly followed the VR cams or was in the cab, a stereo and a mono mic externally, and a series of mono spot sources mounted on the truck.   

 

Edited by Wandering Ear
Fixed video link
Link to comment
Share on other sites

I did all the production recording for both the VR video and the Recoil 3 (non VR) video, the sound design was done by Keith Ruggiero, who did a great job.  For the VR, it's a fixed perspective binaural mix, the binaural head was usually underneath or near the VR cams.  Big thanks to the grips, and my colleague Matt Sheldon for helping me rig an 8 pound binaural head safely inside the vehicle. 

I'd love to do a spacial remix of this video, it could be a lot of fun.

Link to comment
Share on other sites

I agree with what most said, that apart from the ambience most of the sound imagery needs to be created in post. My somewhat philosophical approach to VR is that acoustic perception and visual perception are quite different per se: The visual range of sight is quite limited (what was it, some 180 degrees), which is why we can turn our head in order to change the visual perception, the "frame", completely. Acoustic perception is quite different, the human is much more passive in it, and the acoustic world is everywhere around us, 360 degrees in 3D (so is the visual world, but we don't see most of it).  We also have eyelids that allow us to completely shut out any visual input at our convenience, while completely shutting down our ears is practically impossible. So my point is, I guess, that when we see a VR video, we can move around to pick that part of the whole which we want to see, while the acoustic surroundings are always around us and don't change that much by changing our physical position or orientation, so different rules would apply for mixing audio for VR than just going along with the image. I guess we'll still continue to record most sounds in mono, adding 3D/soundfield ambiences to our portfolio.

Link to comment
Share on other sites

K Ruggerio totally rocks--I worked with him on a similar video in the "Gymkhana" series.  Where inside the car were you able to put the binaural head, when the cams saw 360?  Did they paint it out, or did you do some passes with the head not in the car?  Were you able to get them to agree to rigging your gear during pre-pro?

Link to comment
Share on other sites

Inside the car the head was rigged floating above the passenger seat.  I used a 5/8" baby pin screwed into the top of the nueman head, and the grips built a multi point rig to hang the head and attached the camera so it was positioned directly above the head.  In several of the shots you can see both the rigging used inside the car, and the rigging used on the hood, I think the head gets lost in the dark.  The head stayed in the car for the hood position.  Outside the car the head was hanging from a cardalini on the stand with the camera, which can be seen at times.  I'll try to take a screen grab or pull up some photos from production, there is a cool drone shot at the end of the regular video (recoil 3) of the vr cam, head, and recorder on a stand, and the truck doing 360's around it.  His driving is so precise he can plant the inside tire about an inch away from the stand, spin a 360 and never touch it.

Most of the rigging happened the first half of day one (of 3), much of which I was off recording the narrative bits in the woods for, so Matt and the grips stayed with the trucks and rigged.  Funny side note, I had to stuff the ears with over covers to manage the wind.

I did a fair bit of wild recording on that shoot too, both the 360 I described above, as well as me riding inside the truck holding a group of mics outside the truck getting tire squeals, etc. It's one thing to virtually ride inside the truck via the video, but it's nothing compared to feeling the truck take off and spin in circles first hand.  More than the g forces , I was surprised how far the truck leaned in each direction when sliding.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...