MrDeepFakes Forums

Some content may not be available to Guests. Consider registering an account to enjoy unrestricted access to guides, support and tools

  • We are looking for community members who are intested in helping out. See our HELP WANTED post.

SAE / Question for @iperov

lackeyproton

DF Vagrant
Hi @iperov, thanks for pushing hard for SAE in your comments.

I initially used H128 and then just kept using it because I didn't want to spend more cycles training a brand new model. When I finally got around to it, I was blown away by how much better SAE was - I totally understand why you considered removing the other models as options. It's been so good that I am thinking of writing a SAE guide in the Guides section when I get a chance.

Had a couple of questions for you:

Code:
1) Is there a way to tell the extractor to just use the existing "aligned_debug" images? I was performing a delete/manual fix of approx. 5000 images, but while in the middle of the process, the "aligned" folder got corrupted and I had to delete its contents. 

I made a copy of my "aligned_debug" folder, then I tried various commands / options on the extractor to try and have it use the existing images, but it seems to always want to delete the "aligned_debug" directory and wants to scan the faces on its own every time.

I tried running the full-manual extraction, and then copying in my "aligned_debug" images once the program has begun the manual extraction process, then skipping all the images to see if it would use my my manual alignments. That didn't work either.

Would changing the python code be the only way to preserve all the alignment work I've done so far? I write software so if it's a matter of altering a bit of code I can do that.

Code:
2) In the SAE model, what does the pixel loss flag actually do? I know that you recommend turning it on if the DST face is masked, like your Cage / Magneto example. Does the same apply for a blowjob scene, which causes random obstructions around the face? Is it better to train 15-20K epochs without pixel loss, and then turning pixel loss back on for that situation? Or would you recommend it just stay off for the whole training?

Thanks for any questions you answer. I'll be sure to collect the information into a future guide.
 

lackeyproton

DF Vagrant
I wrote code to achieve what I wanted in question 1 - I manipulated input_path_image_paths in Extractor.py and created an argument --re-extract-only that skips the aligned_debug directory deletion.

So please disregard that one. If you could elaborate on 2, that'd be great! Thanks.

Edit: Ha, I may have spoken too soon, as it looks like there is more to it than that - starting to think that 1) is may not be possible actually, because it looks like the _debug output is the result of face data, not the other way around.

It seems like it all has to stay in memory at once before the extractions are finalized. So I think it looks like the aligned_debug images are a result of the face data in memory. and not the source from which the face alignments are derived.

So I guess I've lost a good deal of manual alignment work by losing the aligned folder, but I guess I learned something. :)
 

mondomonger

DF Admirer
Verified Video Creator
I'm struggling with SAE when it comes to obstructions in the face. With H128, you can achieve decent BJ vids. At a good angle, when the dst pornstar sticks her tongue out, the tongue still shows on the H128 face replacement. SAE seems to treat these obstructions differently. I end up with a warped face where the tongue is replaced with a morphed (larger) face. Blowjob portions of videos for the most part do not look good with SAE.

I'm still early in the process of creating content with SAE models, but so far.....

- SAE color match is WAY better than H128. Really superb!
- SAE handles profiles better, but in some 3/4 and low angle views, SAE is worse than H128. It looks like SAE replaces half the face.
- SAE does not like facial obstructions at all (hands, hair, dicks, etc). (I do not use these obstructions in the training dst set)

-H128 has color match issues. (I have more color problems than I did with Fakeapp 2.2 for example)
-H128 can handle tongue out, dick in mouth, fingers in mouth. There is some distortion, but the illusion of the fake is still acceptable.

If you are doing softcore, masturbation, JOI type vids, SAE is the far superior choice. If you want hardcore porn with blowjobs, tongue out cumshots, etc, then H128 is so far the better choice. My wish would be for the H128 color match to be improved.

I'll post 2 hardcore SAE vids in a few hours that I hope will show the differences between SAE vs H128. They are my first try at this and the clip kind of tests the boundaries a bit in terms of angles, profiles, etc.
 

iperov

DF Enthusiast
Developer
@mondomonger

- SAE does not like facial obstructions at all (hands, hair, dicks, etc). (I do not use these obstructions in the training dst set)

why are you posting disinformation?

SAE_Cage_1.jpg

Hr5Ollm.jpg

52581024-14e8d300-2e43-11e9-9bf1-aabd52c05cce.jpg
 

frosty3907

DF Admirer
Verified Video Creator
iperov said:
@mondomonger

- SAE does not like facial obstructions at all (hands, hair, dicks, etc). (I do not use these obstructions in the training dst set)

why are you posting disinformation?

SAE_Cage_1.jpg

Hr5Ollm.jpg

52581024-14e8d300-2e43-11e9-9bf1-aabd52c05cce.jpg

I wouldn't say disinformation - the magneto mask example seems to be an anomaly, perhaps it deals with the edge of the mask being eroded better. But in all the other examples you can see it's trying to replace the obstruction with the face itself, which mirrors my (and many others) results - there's been many examples of obstructions just been completely occluded by the underlying face.
 

halo2k

DF Pleb
Verified Video Creator
Well @"iperov" Iperov he is right.

Trained an SAE model for several days where the girl had half of her face covered with hair and what SAE tried was, reconstruct the entire half missing face. The result was half transparent hair with very strange looking face.

There are tons of open Questions, same as here:
https://mrdeepfakes.com/forums/thread-sae-vs-h128?pid=2636#pid2636

Please consider taking a few minutes to answer them! Would really appreciate it!


Well @"iperov" he is right.

Trained an SAE model for several days where the girl had half of her face covered with hair and what SAE tried was, reconstruct the entire half missing face. The result was half transparent hair with very strange looking face. Same with obstructions like in Bjs...anything that is obstrucing the face gets removed.

There are tons of open Questions, same as here:
https://mrdeepfakes.com/forums/thread-sae-vs-h128?pid=2636#pid2636

Please consider taking a few minutes to answer them! Would really appreciate it!
 

iperov

DF Enthusiast
Developer
man wrote
SAE does not like facial obstructions at all (hands, hair, dicks, etc)
I showed proofs that he is not right.
other man wrote "he is right".
why should I be willing to answer questions then? I have no time for trolls and inadequates.
just dont use sae then.
 

frosty3907

DF Admirer
Verified Video Creator
iperov said:
man wrote
SAE does not like facial obstructions at all (hands, hair, dicks, etc)
I showed proofs that he is not right.
other man wrote "he is right".
why should I be willing to answer questions then? I have no time for trolls and inadequates.
just dont use sae then.

No-one's trolling you man - why would we?

We're just saying that so far what we've seen with SAE is that it's removing obstructions, there's been examples posted and even some of yours show it (the 2nd and 3rd particularly).
It's not necessarily a bad thing, it's preferable in some cases probably but for what this is used for in the vast majority of cases (porn) it's more desirable to have an occlusion mask for obstructions.

So to be clear what we're saying, and we're not trolling- just stating our results- is that if there's an occlusion of the face, SAE is blending it out with the face going over the top of the obstruction.
 

mondomonger

DF Admirer
Verified Video Creator
I would certainly fall in the inadequate category!

Let me put this another way, with an erode of 50, the SAE face replacement goes what looks like a good 8-10 cm below the chin. This wipes out most of the tongue and a good part of the dick. This leads to some odd looking swaps in some positions. I've done 1 video using both methods if anyone is interested to see the differences. It's a challenging video with a lot of profiles and odd angles. Maybe if I go into debug mode and check the alignments I can fix this, but not sure if I have the time/ability for that.

SAE is better than H128 based on the two vids I have tried so far in every single way --- except for BJ / Cumshots where that larger overhanging faceswap that periodically goes past the chin looks strange. This is now down to me learning what to include vs exclude for SAE vids and seeing how fast an established SAE model can be recycled into a new face.

Its fun to try new methods and sharing here helps us all to get better.


You can see what we are talking about at the 3:36 mark in the SAE vs H128 versions of "Angelina Jolie by Pool" that I just posted. For both, using Overlay, no histogram, Erode = 50, Blur = 10.

H128 = Fingers look good going into the mouth. Acceptable for sure.

SAE = Fingers are overwritten by the faceswap. Fingers disappear. Oval faceswap portion extends through the fingers and looks odd.


Question: For training, I normally remove frames with hair/fingers/dicks that obstruct the face. In order for SAE to train properly around the obstructions, should I leave obstructed faces within the data_dst training set?

Many thanks!
 

lackeyproton

DF Vagrant
@"iperov" , thanks for any input you can provide. I really love SAE, so I'd love to keep learning more about it. I would love to hear more about pixel loss and how it can be best used for DFL.

And another thought - is it best to train vids maybe 1-2 minutes at a time for modeling efficiency? Or can one model handle videos as long as 5-6 minutes? Not sure if there's such a thing as "jack of all trades but master of none" for models.
 

mondomonger

DF Admirer
Verified Video Creator
One frame that shows the difference.


SAE 



H128
[size=small]


If you open each image in 2 tabs and then click between the two, you'll also see a facial positioning difference. The SAE faceswap sits slightly more to the upper left while the H128 face is positioned more correctly. I think that facial positioning difference is causing the problem. Both are using the same MT face PNG files and are set to Overlay, no histogram, erode 50, blur 10.

If that SAE swap was positioned in the same exact place as the H128 swap, I think this problem goes away.
 

lackeyproton

DF Vagrant
mondomonger said:
One frame that shows the difference.


SAE 



H128
[size=small]


If you open each image in 2 tabs and then click between the two, you'll also see a facial positioning difference.    The SAE faceswap sits slightly more to the upper left while the H128 face is positioned more correctly.      I think that facial positioning difference is causing the problem.     Both are using the same MT face PNG  files and are set to Overlay, no histogram, erode 50, blur 10.

If that SAE swap was positioned in the same exact place as the H128 swap, I think this problem goes away.


Interesting. By the way, is your SAE architecture LIAE? Or something else? Would that matter at all?
 

frosty3907

DF Admirer
Verified Video Creator
lackeyproton said:
mondomonger said:
One frame that shows the difference.


SAE 



H128
[size=small]


If you open each image in 2 tabs and then click between the two, you'll also see a facial positioning difference.    The SAE faceswap sits slightly more to the upper left while the H128 face is positioned more correctly.      I think that facial positioning difference is causing the problem.     Both are using the same MT face PNG  files and are set to Overlay, no histogram, erode 50, blur 10.

If that SAE swap was positioned in the same exact place as the H128 swap, I think this problem goes away.


Interesting. By the way, is your SAE architecture LIAE? Or something else? Would that matter at all?


yes I'd previously been using liae and am now trying df arch to see if the results are the same.
 

lackeyproton

DF Vagrant
Another thought - isn't placement dictated more by the MT/DLIB face dictation algorithm? Or are you saying that the models behave differently even when presented with the same alignment data?
 

lackeyproton

DF Vagrant
Wanted to let you know that I discovered that the github is probably the best place for the kind of info I was asking about. It's under the "Issues" tab under Free For Talk that iperov set up.
 
Top