MrDeepFakes Forums

Some content may not be available to Guests. Consider registering an account to enjoy unrestricted access to guides, support and tools

  • We are looking for community members who are intested in helping out. See our HELP WANTED post.

Tips for avoiding model collapse?

Kamrooz

DF Vagrant
Just curious what methods are out there for avoiding model collapse. ATM My models always seem to collapse roughly around 100,000 iterations. Not using anything special, mostly default settings although on the higher quality side. Any methods or tips for extending this?

Also another question, does having too many src images do more harm than good? I have around 5700 source images. 

Appreciate any info and advice. Thanks guys ^_^.
 

dpfks

DF Enthusiast
Staff member
Administrator
Verified Video Creator
Higher style power or using pixel loss will increase chances of model collapse.

Having too many data_src can hurt your training time. I generally keep it below 5000 images.
 
I have the same issue after about 100000 iterations too. When I watch previews closely I notice some frames just look weird or won't draw right even though the images are aligned properly and almost every other frame looks fine. I'm noticing that when I train for a while using just those messed up frames they tend to clear up and I'm having less collapsing issues after. Not sure if there's really any connection though.

Having a lot of images adds training time, and also takes up more memory. It depends on the situation. With a lot of them it's easier to get subtle facial expressions and things like that, but again, it depends on the situation whether or not it's worth it to you. One of my projects has 15000+ src images, and I could not get the results I'm getting using less. I tried... maybe I just suck though lol. I usually use between 5000-10000 src and am finding best results this way.
 
So far, new Gradient Clipping feature seems to be helping things. Using pixel loss and not seeing spikes or a collapse yet. Before this update I was having issues every 100-200 iterations with pixel loss on.
 

[deleted]

DF Vagrant
H128 typically doesn't crash even if you reuse models, as long as no pixel loss is used, and its not really needed if you have good facests and train enough. But for SAE, don't reuse models and back up every 15 min if you are using pixel loss
 
GhostTears said:
H128 typically doesn't crash even if you reuse models, as long as no pixel loss is used, and its not really needed if you have good facests and train enough. But for SAE, don't reuse models and back up every 15 min if you are using pixel loss

I was saving every 5 mins lol. 

No collapse or spikes yet using latest version and gradient clipping.  Have gone over 20000 with pixel loss with no problems.
 

Kamrooz

DF Vagrant
I don't even use pixel loss and my models collapse right before 100k. Normally around 95k iterations. That's the part that's confusing the hell out of me. No pixel loss, and no face style power. That's the part that confuses me the most. I'm not even running features know to cause model collapse, and my model still collapses. I haven't used the build with gradient clipping just yet. Going to download that now and give it a go and see what happens. *crosses fingers*
 

dpfks

DF Enthusiast
Staff member
Administrator
Verified Video Creator
Kamrooz said:
I don't even use pixel loss and my models collapse right before 100k. Normally around 95k iterations. That's the part that's confusing the hell out of me. No pixel loss, and no face style power. That's the part that confuses me the most. I'm not even running features know to cause model collapse, and my model still collapses. I haven't used the build with gradient clipping just yet. Going to download that now and give it a go and see what happens. *crosses fingers*

How many images in data_src?

Yes this new feature should hopefully stop your collapse
 

Kamrooz

DF Vagrant
dpfks said:
Kamrooz said:
I don't even use pixel loss and my models collapse right before 100k. Normally around 95k iterations. That's the part that's confusing the hell out of me. No pixel loss, and no face style power. That's the part that confuses me the most. I'm not even running features know to cause model collapse, and my model still collapses. I haven't used the build with gradient clipping just yet. Going to download that now and give it a go and see what happens. *crosses fingers*

How many images in data_src?

Yes this new feature should hopefully stop your collapse

5400 source images. I run with 768 autoencoder dims as I always tend to have problems with eye anomalies, and 256 res...Most other settings are default. High quality source images as well.
 

Kamrooz

DF Vagrant
Have to admit, this new gradient clipping feature is quite impressive. A model that would always collapse around 95,000 iterations (had it saved at 92k), is now on 135k. Quite a helpful feature
 

coveredinnormal

DF Vagrant
GhostTears said:
H128 typically doesn't crash even if you reuse models, as long as no pixel loss is used, and its not really needed if you have good facests and train enough. But for SAE, don't reuse models and back up every 15 min if you are using pixel loss

I used to use only H128 now I use only SAE and I also reuse a lot of models/people/celebs (I have a library) one of my models is in the 250,000 iteration range. The models I have with high numbers train way faster because they have been under the microscope longer. I don't use pixel loss and I have been using a smorgasbord of different settings ideas from various users on this forum. I have learned, as of tonight, not to use over 5,000 src images. That will stop as of now. Also I back my stuff up every 15min. If I have a model collapse I find a good back up and play with settings until I figure out what caused it. Then I keep on going.
 
Top