Mr DeepFakes Forums
  • New and improved dark forum theme!
  • Guests can now comment on videos on the tube.
   
H128 model full degrading with Pixel Loss on?
#1
Hi all,

I'm trying to run pixel loss on an h128 model to get rid of some facial jitters and fine out some details, and it seems that I can't run more than a few hundred epochs without this happening (SRC redacted since it's not a public figure). I'm running a GTX1070 Ti with default h128 options other than pixel loss (batch 16, history off). There was nothing in the cmd window to indicate an error or memory shortage. At some epoch, the loss values were around .09 for src and .03 for dst, and then suddenly they started climbing to the point that the predictions are just flat red.

src is about 1600 images and dst is about 6000 if it matters. Has anyone encountered anything similar and have any suggestions?

[Image: KSDoYW5.jpg]
#2
Unfortunately your model cashed. You need to start over, or I hope you made a backup.

Unsure why it crashed.
#3
(03-15-2019, 05:50 AM)dpfks Wrote: You are not allowed to view links. Register or Login to view.Unfortunately your model cashed. You need to start over, or I hope you made a backup.

Unsure why it crashed.

Yes, thankfully I kept a copy of the model pre-pixel loss. I played around with it some. It seems that it's crashing after running after doing a run of a certain number of epochs as opposed to a particular numbered epock. I was able to train from 478100 to 479000 without a crash in increments of 100, but trying to train from 478100 straight to 479000 crashed after 300-400 epochs repeatedly.

Forum Jump:

Users browsing this thread: 1 Guest(s)