MrDeepFakes Forums

Some content may not be available to Guests. Consider registering an account to enjoy unrestricted access to guides, support and tools

  • We are looking for community members who are intested in helping out. See our HELP WANTED post.

DeepFaceLab SAE training makes eyes look in wrong direction?

honkywonkyman

DF Vagrant
I'm currently at iteration 20 000 training an SAE model on the Google Colab. Overall the training is progressing pretty nicely, but I've noticed that the eyes of the decoded face never seem to look in the same direction as the dest, and I'm worried this will turn into an issue later on in the training if I don't fix it. I also saw this picture of a Robert Downey JR - Elon Musk model, where you can clearly see that Elon's eyes don't match Robert's even though the model is at 85k iterations and the conversion is convincing otherwise:

Is this whole thing an issue with SAE or are there just some settings that need to be changed or something? If it's not SAE, do any of you have any ideas or suggestions on how to solve it?

Here are my training settings:
== Model name: SAE
==
== Current iteration: 19643
==
== Model options:
== |== write_preview_history : True
== |== target_iter : 40000
== |== batch_size : 8
== |== sort_by_yaw : False
== |== random_flip : False
== |== resolution : 128
== |== face_type : f
== |== learn_mask : True
== |== optimizer_mode : 1
== |== archi : df
== |== ae_dims : 512
== |== e_ch_dims : 42
== |== d_ch_dims : 21
== |== multiscale_decoder : True
== |== ca_weights : False
== |== pixel_loss : False
== |== face_style_power : 10.0
== |== bg_style_power : 10.0
== |== apply_random_ct : False
== |== clipgrad : False
== Running on:
== |== [0 : Tesla T4]
=========================


Example of what I mean. The face structure looks right except for the eyes.
XBjCtrch.png
 

dpfks

DF Enthusiast
Staff member
Administrator
Verified Video Creator
Known issue, Don't think theres a fix. Seems like eyes are hard to get right with deepfakes.
 
I will be posting some things shortly, but maybe not here.  I've got the eyes pretty convincing now.  I can maybe do a tutorial or something because it's not just straightforward like do X for Y iterations at Z settings.

I've been considering doing something like that as my methods are different than I see others using, and I am getting close to 100% accuracy with eyes and mouth and subtle facial expressions.
 

honkywonkyman

DF Vagrant
@"TheMadFaker" That would be appreciated. Should I pause training for now until I fix the eyes or can I continue training and fix it later? (yes, I do know it's not that simple)
 

threecaptain

DF Vagrant
Verified Video Creator
You could try raising the style power as that setting morphs the source face to more closely match the dst. Maybe the eyes will follow suit and you can switch back to your original settings.
 
Top