MrDeepFakes Forums
  • New and improved dark forum theme!
  • Guests can now comment on videos on the tube.
   
VR Deepfakes; what's different?
#1
In the last few days I've gotten a few messages from people who want to get started doing VR clips so I thought I'd do a guide and explain to the best of my ability, how the process of making VR differs from regular 2d clips.

keep in mind I only use Deepfacelab.


Making VR deepfakes is exactly the same process with two added steps. Frame splitting before face extraction, then frame merging back to 16:9 stereo picture format, before the final conversion to video. 

1st you'll want to cut the sequence of the VR video you want to work with, using the "cut video" batch file provided with DFL, just like you would do with a regular DF. I usually keep my VR clips under 5-7 minutes. Next you need to extract frames from said clip, as you would do with regular DFs. Then only you're ready to split the frames in your data_dst with this program:

Here's a link for SplitVR2.0, A small utility made by /uCoderMotor. 
You are not allowed to view links. Register or Login to view.
It uses ffmpeg so if you know you're way with it, you might not even need this. 
the program is stand alone and pretty self explanatory.

You want to edit the "Split" batch file so it knows your data_dst directory as well as the "Join" batch file so it knows where your data_dst/merged folder will be. That's pretty much all there is to it. The program will split every frame you have in a given folder and rename them properly with a "L" or "R" prefix

Once you've completed the splitting process you will end up with a backup folder in data_dst that you can either delete or keep if you want. At that point you're ready to exact/align faces.

Everything you know about 2d deepfakes also applies in VR. You want the face to be as stable as possible and make sure the angles will translate well when training. VR videos can have bad sequences where the faces are really warped. I usually skip those parts altogether. When the faces are right in the center of the frame, they wont be warped at all. I know there is a program called "VRripReplace" that can "unwarp" VR spherical projection but it seemed pretty tedious to make it work with DFL the last time I checked. I'm interested to hear about your experiences with it, if you have tried it.

Also, you might get better results when training one channel at a time, probably even more so when working with low VRAM(?) but I usually do L & R at the same time. 

Once you've completed training and converting just as you would do with 2d DFs, right before converting to mp4, you need to merge back all your converted frames with the "join" batch file (be sure it knows where your data_dst/merged folder is) 

then, convert to mp4 and you're ready to go! 

As you can see, it's not much more complicated, It's just more time consuming, but the results will be worth it!

I hope this comes handy to some of you!
#2
Thank you! This should be pinned thread
#3
This post has been pinned in the Guides section.
#4
Really good guide, and cool guy!
Thanks
#5
I'm having an issue with merging left/right. It just wont do it.

First time I did it, it merged till 100%.... but then the merged images are nowhere to be found and now it wont do it again.

Can anyone help me?

Forum Jump:

Users browsing this thread: 2 Guest(s)