MrDeepFakes Forums

Some content may not be available to Guests. Consider registering an account to enjoy unrestricted access to guides, support and tools

  • We are looking for community members who are intested in helping out. See our HELP WANTED post.

[GUIDE] [LEGACY] [DFL 1.0] Training FANSEG model in DFL 2.0

CapHarlock

DF Pleb
Hi,

While waiting for time to make a more detailed guide, I leave you here the basic steps so you can train your own FANSEG model and get good results with the obstructions when making merge, especially in the final parts of your videos ("If you know what i mean", lol).


First of all you have to know that, to make the process a little easier I have decided to use a modified version of the .bat file that executes the process; basically to have a FANSEG folder where you can train this model and that doesn't interfere with the "workspace" folder.

Create (or modify if you already have it) a .bat file called "6)Train FANSEG 2.0" and copy this code inside:
Code:
@echo off
call _internal\setenv.bat

"%PYTHON_EXECUTABLE%" "%DFL_ROOT%\main.py" train ^
   --training-data-src-dir "FANSEG\data_masked\aligned" ^
   --training-data-dst-dir "FANSEG\data_masked\aligned" ^
   --model-dir "FANSEG\model" ^
   --model FANSeg

pause
(This file must be in the root folder, at the same level where you have the other bat files)


If you don't want to start your FANSEG model from scratch you can use mine.
It contains the original/default FANSEG as well as 550 additional trained masks (with "NSFW material"):
https://drive.google.com/open?id=13dUCKyNA5nPppu1_GWZVgwf-vavc1Sm5



Steps to follow:
- Create a directory called "FANSEG" at the same level where you have "workspace" (or _internal).
- OPTIONAL: Copy my model there if you want to keep training from it.

- Prepare your video "data_dst.mp4" as usual, extracting the frames and aligning the faces.
- Use "Mask Editor" in the final frames (or those where applicable) to modify the masks and "remove" parts such as the inside of the mouth, tongue, co__, hands, cu__, ... Save the changes with "e".
- The images with modified masks will have been moved to "aligned_confirmed", so they should be copied (not moved) to "aligned" in order that all the images are there again.
- Train your SAEHD model (workspace directory) as you normally would until the result is good enough.
- Copy the images in ".../workspace/data_dst/aligned_confirmed" to ".../FANSEG/data_masked"
- Run the modified .bat file to train FANSEG ( 6)Train FANSEG 2.0.bat )
- Save the model when you see that the mask has been learned (it should not take more than 1000-2000 iterations).
Copy your FANSEG model into ".../_internal/DeepFaceLab/facelib" and rename it to "FANSeg_256_full_face.npy" (make a backup of the original just in case).
Merge with FAN-X and check the result (if not satisfactory train your FANSEG with some additional masks)


I hope this will facilitate the process and help to understand it better.



PS: If you want to see how the masks have been modified for training with FANSEG here are some examples:
https://drive.google.com/open?id=1Hw4Kmbi66DdYF7mS9O6YywFXZGMlaK7A

PS2: As soon as I have some time I will extend the guide making it more detailed and with some images (although I don't know if this forum section allows NSFW material :p)


Enjoy your fakes!
 

titan_rw

DF Pleb
I'm kind of confused on your work instructions.

Are you editing the mask on all frames that have obstructions?  Because if you are, there's no need to train a fanseg model.

If you aren't, it doesn't seem like you're setting up the training directories properly for fanseg training.

Quote from iperov on github:

data_src is used for training the network
data_dst is the second preview to test the network on unseen images

You're pointing src and dst to the same set of faces.
 

CapHarlock

DF Pleb
titan_rw said:
I'm kind of confused on your work instructions.

Are you editing the mask on all frames that have obstructions?  Because if you are, there's no need to train a fanseg model.

It's not necessary to "create masks" for each frame, of course, they should only be created in some frames that show the obstructions you want the FANSEG model to learn, in their different forms (ex.: tongue out front/sides/etc; inside the mouth from different angles, etc...). In my tests, with videos of between 700-1000 frames (final part of the video) there has been enough with between 20-40 samples to get an excellent result.


titan_rw said:
If you aren't, it doesn't seem like you're setting up the training directories properly for fanseg training.

Quote from iperov on github:

data_src is used for training the network
data_dst is the second preview to test the network on unseen images

You're pointing src and dst to the same set of faces.

This is because, for ease of use, I have modified the .bat file and changed the folders that are used to train FANSEG; basically for two reasons:
- If I didn't modify it, I needed to always have a folder "workspace_FANSEG" (for example), and the usual "workspace"; then to train FANSEG I had to rename "workspace" to "workspace_MODEL" (for example), and "workspace_FANSEG" to "workspace" to train FANSEG; and again invert the names to make the final merge... it was an unnecessary process; now you have a folder called "FANSEG" where to train your model and you will never have to change the name :p
- As iperov says, when training FANSEG, the "data_dst" folder is just a preview to check the result; it's not really important, and you can use the same images that would be in "data_src"; so I decided to modify the .bat file and use a single folder called "data_masked" that is used both as "data_src" and for "data_dst" :-D



NOTE: As soon as I have some time I will make a more visual guide with screenshots to make it easier to understand... although I may not be able to create it for a couple of weeks.
 

titan_rw

DF Pleb
We'll agree to disagree then. I think it's important one way or another to have distinct src / dst faces for fanseg training.

If you use nothing but your edited mask faces for both, then you can't really see how the model is progressing. It will always be perfect at masking images it's seen and trained on already. But you don't know how it's doing on unseen pictures.

If you put every face (included non mask-edited) into the src and dst directory, then it'll train way slower as most of the images you're training on don't have the obstruction masked out. You also still have already trained images being used for 'predicted' output.

For me, to keep it simple, I just use a separate DFL directory when I need to train a custom fanseg model. I just name it c:\DFL-SEG\ or something. Put mask edited files in src, and the rest (unseeen) in dst. Then train, and copy model to main c:\DFL\ directory when done.
 

CapHarlock

DF Pleb
But FANSEG must only train the mask, and you can see that progress in the preview even if you have the same images in dst and src, you have to look at the black and white mask and train it until the edges are defined.

However, if you wish, you can also modify the .bat to have another folder for the "dst":
--training-data-dst-dir "FANSEG\data_unmasked\aligned
 

titan_rw

DF Pleb
CapHarlock said:
But FANSEG must only train the mask, and you can see that progress in the preview even if you have the same images in dst and src, you have to look at the black and white mask and train it until the edges are defined.

That's exactly my point.  All you're seeing is training result of data that's is already trained on.  Of course it's going to be fine.  It's told "this image should be masked 'this' way".  How could it get that wrong?

What you care about is images it hasn't seen.  How well does it do (masking) with an image that hasn't had the mask hand specified?  That's why DST should be unseen (unmasked) faces.  The entire point of training a custom fanseg is that it performs well on all images, especially those the fanseg model hasn't seen before.  By not including those, you don't really know how well it's performing.

By seeing the 'predicted' mask on unseen images you can tell how well the model is working over the entire project.  The whole point is to not need to hand mask every single images.  So why not 'see' the resulting mask of the images you haven't hand masked?  I've seen DST images where the mask isn't working, and I had to go back and hand mask a couple extra images so that it got the mask right on everything.  But of course all the SRC masks all looked fine, because they're all 'known' to the model.

Again, let's agree to disagree.  It's easier.
 

CapHarlock

DF Pleb
titan_rw said:
CapHarlock said:
But FANSEG must only train the mask, and you can see that progress in the preview even if you have the same images in dst and src, you have to look at the black and white mask and train it until the edges are defined.

That's exactly my point.  All you're seeing is training result of data that's is already trained on.  Of course it's going to be fine.  It's told "this image should be masked 'this' way".  How could it get that wrong?

What you care about is images it hasn't seen.  How well does it do (masking) with an image that hasn't had the mask hand specified?  That's why DST should be unseen (unmasked) faces.  The entire point of training a custom fanseg is that it performs well on all images, especially those the fanseg model hasn't seen before.  By not including those, you don't really know how well it's performing.

By seeing the 'predicted' mask on unseen images you can tell how well the model is working over the entire project.  The whole point is to not need to hand mask every single images.  So why not 'see' the resulting mask of the images you haven't hand masked?  I've seen DST images where the mask isn't working, and I had to go back and hand mask a couple extra images so that it got the mask right on everything.  But of course all the SRC masks all looked fine, because they're all 'known' to the model.

Again, let's agree to disagree.  It's easier.

I don't see the need to have "unmasked" images for verification; if at the time of making merge you don't see that you have masked enough images simply make more.
FANSEG is very powerful and with few samples there is enough, the important thing is to have a variety of samples, with different angles or shapes.
I have used "my system" (to say it in a way) recently in a type of hair that covered the eyebrows and the result has been really great... and for the tongues and the inside of the mouth it is simply stunning; the most complicated part... well, you know what it is ;-)

Anyway, as I told you, I just modified the .bat a little bit to make it easier and faster, but you can always use the "original method".
 

tomguluson92

DF Vagrant
Hi buddy, thanks for your great tutorial in handling occlusions.

I just wonder which version do you use? Since 2020.03.07 version does not contain the FAN-X merge option?


My interactive merger just like below: there are no FAN-X merger in DFL 2.0.

Or you use DFL 1.0 instead?

wvuEmbSh.jpg
 

CapHarlock

DF Pleb
tomguluson92 said:
Hi buddy, thanks for your great tutorial in handling occlusions.

I just wonder which version do you use? Since 2020.03.07 version does not contain the FAN-X merge option?


My interactive merger just like below: there are no FAN-X merger in DFL 2.0.

Or you use DFL 1.0 instead?

wvuEmbSh.jpg





Last version with FANSEG is 03_27_2020

You can active FAN-X with "x" key (mask mode)  ;)
 

terra

DF Vagrant
Could you upload the tutorial video on YouTube, please?
There are many guys who are interested in that FANSEG way with screen-guide, I guess.
 

CapHarlock

DF Pleb
terra said:
Could you upload the tutorial video on YouTube, please?
There are many guys who are interested in that FANSEG way with screen-guide, I guess.


Yeah, I should have done it, but I never found the time, even though I had some images already prepared.

Although for NSFW stuff I don't think youtube is the most suitable, lol.

I still don't have the time to do it, plus I don't know if it's really necessary right now since it's already been replaced by the new X-Seg... although I recognize that FANSEG was great and in terms of cums it had its advantages :p
 
Top