MrDeepFakes Forums
  • New and improved dark forum theme!
  • Guests can now comment on videos on the tube.
   
  •  Previous
  • 1
  • ...
  • 30
  • 31
  • 32(current)
  • 33
  • 34
  • ...
  • 36
  • Next 
DeepFaceLab - Google Colab Tutorial
im new to see, do the training stop by itself or do i have to stop it? thank you
Hi.

How to enable multiscale decoder option when working in collab?
(09-10-2019, 11:29 PM)DarkSonic22 Wrote: You are not allowed to view links. Register or Login to view.hey all, I'm running into a problem when using Colab.
I did the entire process, from extracting images from the videos to training the model, all on Colab. But when converting, this is what happens:



Code:
/content
Running converter.

Loading model...
Using TensorFlow backend.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
========== Model Summary ==========
==                               ==
==         Model name: SAE       ==
==                               ==
==  Current iteration: 140889    ==
==                               ==
==-------- Model Options --------==
==                               ==
==         batch_size: 12        ==
==        sort_by_yaw: False     ==
==        random_flip: False     ==
==         resolution: 128       ==
==          face_type: f         ==
==         learn_mask: True      ==
==     optimizer_mode: 1         ==
==              archi: df        ==
==            ae_dims: 512       ==
==          e_ch_dims: 42        ==
==          d_ch_dims: 21        ==
== multiscale_decoder: True      ==
==         ca_weights: False     ==
==         pixel_loss: False     ==
==   face_style_power: 0.0       ==
==     bg_style_power: 0.0       ==
==    apply_random_ct: True      ==
==           clipgrad: True      ==
==                               ==
==--------- Running On ----------==
==                               ==
==       Device index: 0         ==
==               Name: Tesla K80 ==
==               VRAM: 11.00GB   ==
==                               ==
===================================
Choose mode:
(0) original
(1) overlay
(2) hist-match
(3) seamless2
(4) seamless
(5) seamless-hist-match
(6) raw-rgb
(7) raw-rgb-mask
(8) raw-mask-only
(9) raw-predicted-only
Default: 1 :
1
Choose mask mode:
(1) learned
(2) dst
(3) FAN-prd
(4) FAN-dst
(5) FAN-prd*FAN-dst
(6) learned*FAN-prd*FAN-dst
?:help Default: 1 : 6
Choose erode mask modifier [-400..400] (skip:0) :
0
Choose blur mask modifier [-400..400] (skip:0) :
0
Choose motion blur power [0..100] (skip:0) :
0
Choose output face scale modifier [-50..50] (skip:0) :
0
Apply color transfer to predicted face? Choose mode ( rct/lct/ebs skip:None ) : ebs
Choose sharpen mode:
(0) None
(1) box
(2) gaussian
?:help Default: 0 :
0
Choose super resolution mode:
(0) None
(1) RankSRGAN
?:help Default: 0 :
0
Degrade color power of final image [0..100] (skip:0) :
0
Export png with alpha channel of the mask? (y/n skip:n) :
n

Collecting alignments: 100% 1988/1988 [00:02<00:00, 912.68it/s]
Computing motion vectors: 0it [00:00, ?it/s]
No frames to convert in input_dir.
Done.




What does "no frames to convert in input_dir" mean? And how can I get around it?

Thanks


Did you by any chance delete images from the workspace/data_dst folder to save space?

(09-11-2019, 01:08 AM)banginpadr Wrote: You are not allowed to view links. Register or Login to view.im new to see, do the training stop by itself or do i have to stop it? thank you

It will stop automatically after 11 hours
(09-11-2019, 08:19 PM)Lord of Calamity Wrote: You are not allowed to view links. Register or Login to view.Did you by any chance delete images from the workspace/data_dst folder to save space?

Nope, it's all there
(08-24-2019, 11:13 AM)blaki Wrote: You are not allowed to view links. Register or Login to view.How to stop the training in colab ? 

" Press "Enter" to stop training and save model. " , This is is not working. Pressing enter key is not doing anything at all, the training is still going on.
Hope you guys will help out on this .

Thanks

Just click the stop button on Google Colab on the top left.

(08-25-2019, 11:59 AM)nargargar Wrote: You are not allowed to view links. Register or Login to view.Hey guys, my google drive is close to being full now (their max is storing 15gb for free) and im only at about 25,000 iterations. I'm shooting for at least 120,000 iterations but if i train any further, the zip file size will pass their 15gb limit.

What should i do?

Use a smaller video, use jpgs and not png for faceset, or just use colab for training.

(08-29-2019, 09:29 PM)ginogini Wrote: You are not allowed to view links. Register or Login to view.I'm using a lot Colab and now I can just find TESTA K80. My question is: how is it possible that my NVIDIA 1660 with 6GB VRAM is faster than a TESLA K80 with 11GB VRAM? It shouldn't be faster the GPU with more VRAM?

VRAM doesn't mean speed, it just allows your to run higher settings.


(09-11-2019, 01:08 AM)banginpadr Wrote: You are not allowed to view links. Register or Login to view.im new to see, do the training stop by itself or do i have to stop it? thank you

Can't. Read the first post.

(09-11-2019, 06:52 PM)visant76 Wrote: You are not allowed to view links. Register or Login to view.Hi.

How to enable multiscale decoder option when working in collab?

You can only do this on the first run of training.

(09-11-2019, 09:11 PM)DarkSonic22 Wrote: You are not allowed to view links. Register or Login to view.
(09-11-2019, 08:19 PM)Lord of Calamity Wrote: You are not allowed to view links. Register or Login to view.Did you by any chance delete images from the workspace/data_dst folder to save space?

Nope, it's all there

Your file structure is probably wrong, or images are missing.
~ Fake it till you make it ~
You are not allowed to view links. Register or Login to view.
You are not allowed to view links. Register or Login to view.
Can't continue my SAE training on colab, help:

Code:
Error: You are trying to load a weight file containing 18 layers into a model with 16 layers.
Traceback (most recent call last):
 File "/content/DeepFaceLab/mainscripts/Trainer.py", line 49, in trainerThread
   device_args=device_args)
 File "/content/DeepFaceLab/models/ModelBase.py", line 169, in __init__
   self.onInitialize()
 File "/content/DeepFaceLab/models/Model_SAE/Model.py", line 344, in onInitialize
   loaded, not_loaded = self.load_weights_safe(not_loaded)
 File "/content/DeepFaceLab/models/ModelBase.py", line 422, in load_weights_safe
   model.load_weights(filename)
 File "/usr/local/lib/python3.6/dist-packages/keras/engine/network.py", line 1166, in load_weights
   f, self.layers, reshape=reshape)
 File "/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py", line 1030, in load_weights_from_hdf5_group
   str(len(filtered_layers)) + ' layers.')
ValueError: You are trying to load a weight file containing 18 layers into a model with 16 layers.

Had no problem training it to 68k iters, suddenly it gives this error...

edit: it gives the same error with another model from a week ago that I've postponed training. Is it a collab/"DFL on collab" problem?
@dozaler
If you used multiscale decoder, then I have bad news for you.
iperov removed this option from SAE a few hours ago.
This is not a colab error.
You need to retrain model.

In theory, you can try to create new model and then replace SAE_decoder.h5 in your trained model to new created.
I'm not sure if this will work.
But if so, it will save you a lot of time.

You can just delete SAE_decoder.h5 and continue training.
New file without multiscale decoder will be generated automatically by DFL
(09-13-2019, 11:51 AM)chervonij Wrote: You are not allowed to view links. Register or Login to view.@dozaler
If you used multiscale decoder, then I have bad news for you.
iperov removed this option from SAE a few hours ago.
This is not a colab error.
You need to retrain model.

In theory, you can try to create new model and then replace SAE_decoder.h5 in your trained model to new created.
I'm not sure if this will work.
But if so, it will save you a lot of time.

You can just delete SAE_decoder.h5 and continue training.
New file without multiscale decoder will be generated automatically by DFL

Thanks for replying. Tho I'm confused with the proposed workarounds; the first one suggests SAE_decoder.h5 holds the trained data and the second one suggests that it does not. And I see there are 4 decoder files: SAE_decoder_dst.h5, SAE_decoder_dstm.h5, SAE_decoder_src.h5 and SAE_decoder_srcm.h5. 

So to clarify which files hold the 68k iters of trained data? and which file/s hold the multiscale decoder?

Also, seeing that I couldn't continue training my previously started models, I started a new SAE training and chose multiscale decoder on again and it has succesfully started training... at 3k iters atm :/ Now I stopped and restarted training with no errors with multiscale decoder: on.
@dozaler

All "decoder" files store a model with multiscale decoder. I'm not completely sure about "_dstm.h5" and "_srcm.h5" because these are mask decoders.

I did not quite understand how you could choose multiscale decoder in the options if iperov removed it from the model. When did your Colab session begin?
lol decoder.h file delete is not solution for me.
All decoder file delete, then training start.
But all training result are delete.
  •  Previous
  • 1
  • ...
  • 30
  • 31
  • 32(current)
  • 33
  • 34
  • ...
  • 36
  • Next 

Forum Jump:

Users browsing this thread: AllThemFakes, 2 Guest(s)