MrDeepFakes Forums
  • New and improved dark forum theme!
  • Guests can now comment on videos on the tube.
   
  •  Previous
  • 1
  • ...
  • 70
  • 71
  • 72(current)
  • 73
  • 74
  • ...
  • 180
  • Next 
(outdated) [GUIDE] - DeepFaceLab 1.0 EXPLAINED AND TUTORIALS
(03-26-2019, 09:07 AM)VFX Wrote: You are not allowed to view links. Register or Login to view.Hi 

Thanks for this amazing app and great source of info.

I'm having a problem at the convert stage and I can't see it covered in this thread. Everything seems to be working perfectly, but when I try to convert my trained SEA model I get the following error. 

It usually converts somewhere between 5-100 images before it fails like this:

Code:
Running converter.

Loading model...
Using plaidml.keras.backend backend.
INFO:plaidml:Opening device "opencl_amd_gfx901.0"
===== Model summary =====
== Model name: SAE
==
== Current iteration: 2128
==
== Model options:
== |== write_preview_history : True
== |== batch_size : 4
== |== sort_by_yaw : False
== |== random_flip : True
== |== resolution : 64
== |== face_type : f
== |== learn_mask : True
== |== optimizer_mode : 1
== |== archi : df
== |== ae_dims : 512
== |== e_ch_dims : 42
== |== d_ch_dims : 21
== |== d_residual_blocks : False
== |== remove_gray_border : False
== |== multiscale_decoder : False
== |== pixel_loss : False
== |== face_style_power : 0.0
== |== bg_style_power : 0.0
== Running on:
== |== [0 : Advanced Micro Devices, Inc. gfx901 (OpenCL)]
=========================
Choose mode: (1) overlay, (2) hist match, (3) hist match bw, (4) seamless, (5) raw. Default - 4 :
4
Suppress seamless jitter? [ y/n ] (?:help skip:n ) :
n
Seamless hist match? (y/n skip:n) :
n
Mask mode: (1) learned, (2) dst, (3) FAN-prd, (4) FAN-dst (?) help. Default - 1 :
1
Choose erode mask modifier [-200..200] (skip:0) :
0
Choose blur mask modifier [-200..200] (skip:0) :
0
Choose seamless erode mask modifier [-100..100] (skip:0) :
0
Choose output face scale modifier [-50..50] (skip:0) :
0
Apply color transfer to predicted face? Choose mode ( rct/lct skip:None ) :
None
Degrade color power of final image [0..100] (skip:0) :
0
Export png with alpha channel? (y/n skip:n) :
n

INFO:plaidml:Analyzing Ops: 123 of 172 operations complete
Collecting alignments: 100%|#####################################################| 5028/5028 [00:03<00:00, 1257.28it/s]
Running on CPU0.
Running on CPU1.
Running on CPU2.
Running on CPU3.
Running on CPU4.
Running on CPU5.
Running on CPU6.
Running on CPU7.
Running on CPU8.
Running on CPU9.
Running on CPU10.
Running on CPU11.
Running on CPU12.
Running on CPU13.
Running on CPU14.
Running on CPU15.
Running on CPU16.
Running on CPU17.
Running on CPU18.
Running on CPU19.
Converting:   0%|                                                                  | 5/5036 [00:59<16:43:36, 11.97s/it]CPU1 doesnt response, terminating it.
CPU3 doesnt response, terminating it.
CPU6 doesnt response, terminating it.
CPU7 doesnt response, terminating it.
CPU9 doesnt response, terminating it.
CPU10 doesnt response, terminating it.
CPU11 doesnt response, terminating it.
CPU12 doesnt response, terminating it.
CPU13 doesnt response, terminating it.
CPU14 doesnt response, terminating it.
CPU15 doesnt response, terminating it.
CPU16 doesnt response, terminating it.
CPU17 doesnt response, terminating it.
CPU18 doesnt response, terminating it.
CPU19 doesnt response, terminating it.
CPU0 doesnt response, terminating it.
CPU2 doesnt response, terminating it.
CPU4 doesnt response, terminating it.
CPU5 doesnt response, terminating it.
CPU8 doesnt response, terminating it.
Converting:   0%|                                                                  | 5/5036 [02:00<33:33:53, 24.02s/it]
Done.
Press any key to continue . . .

Full spec

Imac Pro with Windows 10 running on a partition.
AMD Radeon Pro Vega 64 graphics processor with 16GB of HBM2 memory
10-Core 3.0GHz Intel Xeon W Turbo Boost up to 4.5GHz 23.75MB cache
64GB RAM

Any help hugely appreciated

VFX

I had that issue for a bit too. I had to restart my computer and reinstall deepfacelab. What version are you using? The old SAE models will not work with the newest DFL version.
~ Fake it till you make it ~
You are not allowed to view links. Register or Login to view.
You are not allowed to view links. Register or Login to view.
(03-26-2019, 11:38 PM)bossanova Wrote: You are not allowed to view links. Register or Login to view.
(03-26-2019, 07:09 PM)Turtlepik Wrote: You are not allowed to view links. Register or Login to view.Hi guys. Great app, and super happy that it now supports AMD chipsets. However, I find that when training using my onboard Radeon R5, my GPU usage stays at practically zero. It occasionally jumps up to around 7%, but this is very shortlived. It still trains much faster than anything I've got out of my CPU using MyFakeApp, but I was wondering if this was expected behaviour, or if something is somehow throttling my GPU. All drivers and the BIOS are up-to-date, but I can't find some info on people's experiences of using AMD chipsets yet. Any guidance would be appreciated.

On another note, is it possible to get decent results using a resolution of 64? My models never seem to get past a certain level of blurriness. Do I need to start again with a higher res setting, or is it just a case of waiting and letting the trainer do its stuff?

Thanks in advance.  
I was curious about this myself.  While I was looking for an answer to another question, I found that I couple people weighed in and said that you really need to get between the 50-100k threshold, minimum to lose the blur.

Thanks for the reply. I shall persevere with it then, and see what happens.
(03-27-2019, 12:33 AM)dpfks Wrote: You are not allowed to view links. Register or Login to view.
(03-26-2019, 09:07 AM)VFX Wrote: You are not allowed to view links. Register or Login to view.Hi 

Thanks for this amazing app and great source of info.

I'm having a problem at the convert stage and I can't see it covered in this thread. Everything seems to be working perfectly, but when I try to convert my trained SEA model I get the following error. 

It usually converts somewhere between 5-100 images before it fails like this:

Code:
Running converter.

Loading model...
Using plaidml.keras.backend backend.
INFO:plaidml:Opening device "opencl_amd_gfx901.0"
===== Model summary =====
== Model name: SAE
==
== Current iteration: 2128
==
== Model options:
== |== write_preview_history : True
== |== batch_size : 4
== |== sort_by_yaw : False
== |== random_flip : True
== |== resolution : 64
== |== face_type : f
== |== learn_mask : True
== |== optimizer_mode : 1
== |== archi : df
== |== ae_dims : 512
== |== e_ch_dims : 42
== |== d_ch_dims : 21
== |== d_residual_blocks : False
== |== remove_gray_border : False
== |== multiscale_decoder : False
== |== pixel_loss : False
== |== face_style_power : 0.0
== |== bg_style_power : 0.0
== Running on:
== |== [0 : Advanced Micro Devices, Inc. gfx901 (OpenCL)]
=========================
Choose mode: (1) overlay, (2) hist match, (3) hist match bw, (4) seamless, (5) raw. Default - 4 :
4
Suppress seamless jitter? [ y/n ] (?:help skip:n ) :
n
Seamless hist match? (y/n skip:n) :
n
Mask mode: (1) learned, (2) dst, (3) FAN-prd, (4) FAN-dst (?) help. Default - 1 :
1
Choose erode mask modifier [-200..200] (skip:0) :
0
Choose blur mask modifier [-200..200] (skip:0) :
0
Choose seamless erode mask modifier [-100..100] (skip:0) :
0
Choose output face scale modifier [-50..50] (skip:0) :
0
Apply color transfer to predicted face? Choose mode ( rct/lct skip:None ) :
None
Degrade color power of final image [0..100] (skip:0) :
0
Export png with alpha channel? (y/n skip:n) :
n

INFO:plaidml:Analyzing Ops: 123 of 172 operations complete
Collecting alignments: 100%|#####################################################| 5028/5028 [00:03<00:00, 1257.28it/s]
Running on CPU0.
Running on CPU1.
Running on CPU2.
Running on CPU3.
Running on CPU4.
Running on CPU5.
Running on CPU6.
Running on CPU7.
Running on CPU8.
Running on CPU9.
Running on CPU10.
Running on CPU11.
Running on CPU12.
Running on CPU13.
Running on CPU14.
Running on CPU15.
Running on CPU16.
Running on CPU17.
Running on CPU18.
Running on CPU19.
Converting:   0%|                                                                  | 5/5036 [00:59<16:43:36, 11.97s/it]CPU1 doesnt response, terminating it.
CPU3 doesnt response, terminating it.
CPU6 doesnt response, terminating it.
CPU7 doesnt response, terminating it.
CPU9 doesnt response, terminating it.
CPU10 doesnt response, terminating it.
CPU11 doesnt response, terminating it.
CPU12 doesnt response, terminating it.
CPU13 doesnt response, terminating it.
CPU14 doesnt response, terminating it.
CPU15 doesnt response, terminating it.
CPU16 doesnt response, terminating it.
CPU17 doesnt response, terminating it.
CPU18 doesnt response, terminating it.
CPU19 doesnt response, terminating it.
CPU0 doesnt response, terminating it.
CPU2 doesnt response, terminating it.
CPU4 doesnt response, terminating it.
CPU5 doesnt response, terminating it.
CPU8 doesnt response, terminating it.
Converting:   0%|                                                                  | 5/5036 [02:00<33:33:53, 24.02s/it]
Done.
Press any key to continue . . .

Full spec

Imac Pro with Windows 10 running on a partition.
AMD Radeon Pro Vega 64 graphics processor with 16GB of HBM2 memory
10-Core 3.0GHz Intel Xeon W Turbo Boost up to 4.5GHz 23.75MB cache
64GB RAM

Any help hugely appreciated

VFX

I had that issue for a bit too. I had to restart my computer and reinstall deepfacelab. What version are you using? The old SAE models will not work with the newest DFL version.

I was using the version from the 13th and then I changed to the one from 25th, but I had exactly the same problem with both and I wasn't reusing models.

However I have found a workaround. I installed parallels on my Mac Laptop, which allows me to run windows 10 within the OSX operating system. I plugged in the drive I was using for DFL and hit convert and it worked fine, just ploughed through at 1.5its/sec. So at least I can convert now.

This is the difference I noticed between the two. On the iMac it ran on every one of the 20 available CPUs and failed, on the MacBook Pro it ran on only 2 of the available 8 CPUs and worked fine, so maybe it's overloading the CPU on the iMac Pro. Not an expert in this so thats just a guess.

Edit to add, the upside is I can now use the main machine for training and the laptop for conversions and testing different parameters.
While we're waiting for the tutorial to be updated for SAE, would using the default options for everthing except for batch size and resolution depending on your gpu be a good idea?
(03-27-2019, 03:31 PM)test101101 Wrote: You are not allowed to view links. Register or Login to view.While we're waiting for the tutorial to be updated for SAE, would using the default options for everthing except for batch size and resolution depending on your gpu be a good idea?

yes
Does batch size affect the quality of the final result, or just the speed of processing?

What would be a recommended batch size for a 16gb GPU.

Cheers

VFX
(03-28-2019, 03:11 AM)VFX Wrote: You are not allowed to view links. Register or Login to view.Does batch size affect the quality of the final result, or just the speed of processing?

What would be a recommended batch size for a 16gb GPU.

Cheers

VFX

A lower batch size can converge (basically get as good as it's going to get) faster than a higher batch size, but a higher batch size (when it does converge) should eventually be better trained than a lower batch size. It's kind of dependent on your data sets.

Best results seem to come by starting with a low batch size to get your model to "pretty good" quickly and then crank it up to the maximum your hardware allows to get the max details.

For SAE training, I start with a batch of 4 for 10k iterations, at 10k I turn on face and bg style power to 10, set batch to 8, and run to anywhere between 25k to 40k iterations. Faces are pretty clearly recognizable by that point, but with soft details. At that point I turn on pixel loss, drop face style to .1 and turn it up to the max batch size my card will run.

I do a lot of training on a ShadowPC VM with a 16gb Quadro P5000. On that one, with default settings (128 res, default dims, Optimizer 1) I can run batch 64 after turning pixel loss on with about 3800ms iterations. With the new residual blocks option turned on, my max is about 21 with about a 3500ms iteration time.
(03-28-2019, 04:24 AM)Endalus Wrote: You are not allowed to view links. Register or Login to view.
(03-28-2019, 03:11 AM)VFX Wrote: You are not allowed to view links. Register or Login to view.Does batch size affect the quality of the final result, or just the speed of processing?

What would be a recommended batch size for a 16gb GPU.

Cheers

VFX

A lower batch size can converge (basically get as good as it's going to get) faster than a higher batch size, but a higher batch size (when it does converge) should eventually be better trained than a lower batch size. It's kind of dependent on your data sets.

Best results seem to come by starting with a low batch size to get your model to "pretty good" quickly and then crank it up to the maximum your hardware allows to get the max details.

For SAE training, I start with a batch of 4 for 10k iterations, at 10k I turn on face and bg style power to 10, set batch to 8, and run to anywhere between 25k to 40k iterations. Faces are pretty clearly recognizable by that point, but with soft details. At that point I turn on pixel loss, drop face style to .1 and turn it up to the max batch size my card will run.

I do a lot of training on a ShadowPC VM with a 16gb Quadro P5000. On that one, with default settings (128 res, default dims, Optimizer 1) I can run batch 64 after turning pixel loss on with about 3800ms iterations. With the new residual blocks option turned on, my max is about 21 with about a 3500ms iteration time.
Amazing info thanks!

It seems that a batch size of 64 just crashes my machine, but 32 works ok. I will try your suggested technique.

When running with a batch size of 32 and default settings, I'm getting about 2600ms/iteration. However in task manager it says the GPU is working at around 5% and the CPU around 6%. Is that normal? It's an AMD GPU.
Latest build completely fails no matter what I try on Tesla V100

Quote:Using TensorFlow backend.
Loading: 100%|#############################################################################################################################################################################| 1887/1887 [00:03<00:00, 518.31it/s]
Loading: 100%|#############################################################################################################################################################################| 1739/1739 [00:03<00:00, 532.52it/s]
===== Model summary =====
== Model name: SAE
==
== Current iteration: 0
==
== Model options:
== |== batch_size : 4
== |== sort_by_yaw : False
== |== random_flip : False
== |== resolution : 192
== |== face_type : f
== |== learn_mask : True
== |== optimizer_mode : 1
== |== archi : df
== |== ae_dims : 800
== |== e_ch_dims : 60
== |== d_ch_dims : 50
== |== d_residual_blocks : True
== |== remove_gray_border : True
== |== multiscale_decoder : True
== |== pixel_loss : False
== |== face_style_power : 0.0
== |== bg_style_power : 0.0
== Running on:
== |== [0 : Tesla V100-SXM2-16GB]
=========================
Starting. Press "Enter" to stop training and save model.
2019-03-28 18:10:20.269857: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-03-28 18:10:20.279703: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-03-28 18:10:20.288877: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-03-28 18:10:20.295474: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-03-28 18:10:20.307137: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-03-28 18:10:20.314170: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-03-28 18:10:20.322651: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-03-28 18:10:20.329008: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-03-28 18:10:20.338523: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-03-28 18:10:20.344667: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
Error: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
         [[{{node convolution_3}} = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](convoluti
on_3-0-TransposeNHWCToNCHW-LayoutOptimizer, Const_3)]]
         [[{{node Mean_13/_1213}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name
="edge_9099_Mean_13", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Traceback (most recent call last):
  File "C:\Users\adrian_valentino93\Downloads\DeepFaceLabCUDA10.1AVX_build_03_26_2019\DeepFaceLabCUDA10.1AVX\_internal\DeepFaceLab\mainscripts\Trainer.py", line 93, in trainerThread
    iter, iter_time = model.train_one_iter()
  File "C:\Users\adrian_valentino93\Downloads\DeepFaceLabCUDA10.1AVX_build_03_26_2019\DeepFaceLabCUDA10.1AVX\_internal\DeepFaceLab\models\ModelBase.py", line 362, in train_one_iter
    losses = self.onTrainOneIter(sample, self.generator_list)
  File "C:\Users\adrian_valentino93\Downloads\DeepFaceLabCUDA10.1AVX_build_03_26_2019\DeepFaceLabCUDA10.1AVX\_internal\DeepFaceLab\models\Model_SAE\Model.py", line 375, in onTrainOneIter
    src_loss, dst_loss, = self.src_dst_train (feed)
  File "C:\Users\adrian_valentino93\Downloads\DeepFaceLabCUDA10.1AVX_build_03_26_2019\DeepFaceLabCUDA10.1AVX\_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py", line 2715, in __call__
    return self._call(inputs)
  File "C:\Users\adrian_valentino93\Downloads\DeepFaceLabCUDA10.1AVX_build_03_26_2019\DeepFaceLabCUDA10.1AVX\_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py", line 2675, in _call
    fetched = self._callable_fn(*array_vals)
  File "C:\Users\adrian_valentino93\Downloads\DeepFaceLabCUDA10.1AVX_build_03_26_2019\DeepFaceLabCUDA10.1AVX\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1439, in __call__
    run_metadata_ptr)
  File "C:\Users\adrian_valentino93\Downloads\DeepFaceLabCUDA10.1AVX_build_03_26_2019\DeepFaceLabCUDA10.1AVX\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 528, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
         [[{{node convolution_3}} = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](convoluti
on_3-0-TransposeNHWCToNCHW-LayoutOptimizer, Const_3)]]
         [[{{node Mean_13/_1213}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name
="edge_9099_Mean_13", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Done.
if i conver with the new avx version, i got only 1.5 it/s
if i use the cpu version i got 30 it/s

System: threadripper 16 Core wit 32gb ram an a rtx 2080ti
All drivers up to date
same source and trainig material.
any idea?
  •  Previous
  • 1
  • ...
  • 70
  • 71
  • 72(current)
  • 73
  • 74
  • ...
  • 180
  • Next 

Forum Jump:

Users browsing this thread: 3 Guest(s)