Mr DeepFakes Forums
  • New and improved dark forum theme!
  • Guests can now comment on videos on the tube.
   
Baz4kError with DeepFaceLab
#1
I am pulling my hair out. I get to number 6 and I get errors. I can't figure it out to save my life...I have been searing for 3 days on how to fix this...anyone have any ideas??? I have tried every batch file for 6 and they all do pretty much the same...I have tried installing Python and Cuda...and tried it without them...Any help is appreciated.

Quote:Running trainer.

Loading model...

Model first run. Enter model options as default for each run.
Enable autobackup? (y/n ?:help skip:n) : y
Write preview history? (y/n ?:help skip:n) : y
Choose image for the preview history? (y/n skip:n) : n
Target iteration (skip:unlimited/default) :
0
Batch_size (?:help skip:0) : 0
Feed faces to network sorted by yaw? (y/n ?:help skip:n) : y
Flip faces randomly? (y/n ?:help skip:y) : y
Src face scale modifier % ( -30...30, ?:help skip:0) : 0
Use lightweight autoencoder? (y/n, ?:help skip:n) : n
Use pixel loss? (y/n, ?:help skip: n/default ) : n
Using TensorFlow backend.
Loading: 100%|######################################################################| 559/559 [00:00<00:00, 934.12it/s]
Sorting: 100%|#######################################################################| 64/64 [00:00<00:00, 4936.38it/s]
Loading: 100%|##################################################################| 23610/23610 [00:26<00:00, 876.59it/s]
Sorting: 100%|########################################################################| 64/64 [00:00<00:00, 128.42it/s]
===== Model summary =====
== Model name: H128
==
== Current iteration: 0
==
== Model options:
== |== autobackup : True
== |== write_preview_history : True
== |== batch_size : 4
== |== sort_by_yaw : True
== |== random_flip : True
== |== lighter_ae : False
== |== pixel_loss : False
== Running on:
== |== [0 : GeForce GTX 1080]
=========================
Starting. Press "Enter" to stop training and save model.
2019-08-15 12:59:05.704491: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.707619: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.711253: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.715047: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.730485: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.736032: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.738912: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.742181: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.745097: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.750437: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.754749: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.758638: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:09.601209: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
2019-08-15 12:59:09.642917: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
2019-08-15 12:59:09.645225: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
2019-08-15 12:59:09.649229: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
Error: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
         [[{{node model_1/conv2d_1/convolution}} = Conv2D[T=DT_FLOAT, _class=["locAngrytrain...propFilter"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](training/Adam/gradients/model_1/conv2d_1/convolution_grad/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOptimizer, conv2d_1/kernel/read)]]
         [[{{node loss/model_3_loss/Mean_3/_579}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4418_loss/model_3_loss/Mean_3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Traceback (most recent call last):
  File "C:\Users\bizzy\Desktop\DeepFaceLabCUDA9.2SSE\_internal\DeepFaceLab\mainscripts\Trainer.py", line 107, in trainerThread
    iter, iter_time = model.train_one_iter()
  File "C:\Users\bizzy\Desktop\DeepFaceLabCUDA9.2SSE\_internal\DeepFaceLab\models\ModelBase.py", line 472, in train_one_iter
    losses = self.onTrainOneIter(sample, self.generator_list)
  File "C:\Users\bizzy\Desktop\DeepFaceLabCUDA9.2SSE\_internal\DeepFaceLab\models\Model_H128\Model.py", line 88, in onTrainOneIter
    total, loss_src_bgr, loss_src_mask, loss_dst_bgr, loss_dst_mask = self.ae.train_on_batch( [warped_src, target_src_mask, warped_dst, target_dst_mask], [target_src, target_src_mask, target_dst, target_dst_mask] )
  File "C:\Users\bizzy\Desktop\DeepFaceLabCUDA9.2SSE\_internal\python-3.6.8\lib\site-packages\keras\engine\training.py", line 1217, in train_on_batch
    outputs = self.train_function(ins)
  File "C:\Users\bizzy\Desktop\DeepFaceLabCUDA9.2SSE\_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py", line 2715, in __call__
    return self._call(inputs)
  File "C:\Users\bizzy\Desktop\DeepFaceLabCUDA9.2SSE\_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py", line 2675, in _call
    fetched = self._callable_fn(*array_vals)
  File "C:\Users\bizzy\Desktop\DeepFaceLabCUDA9.2SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1439, in __call__
    run_metadata_ptr)
  File "C:\Users\bizzy\Desktop\DeepFaceLabCUDA9.2SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 528, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
         [[{{node model_1/conv2d_1/convolution}} = Conv2D[T=DT_FLOAT, _class=["locAngrytrain...propFilter"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](training/Adam/gradients/model_1/conv2d_1/convolution_grad/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOptimizer, conv2d_1/kernel/read)]]
         [[{{node loss/model_3_loss/Mean_3/_579}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4418_loss/model_3_loss/Mean_3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Done.
Press any key to continue . . .
#2
refer to this thread. it was the same issue and was solved
You are not allowed to view links. Register or Login to view.
!nvidia-smi
#3
I just tried reinstalling Cuda 9.2 and clearing the workspace and it did the same thing.
#4
(08-16-2019, 03:59 AM)Baz4k Wrote: You are not allowed to view links. Register or Login to view.I just tried reinstalling Cuda 9.2 and clearing the workspace and it did the same thing.

Uninstall CUDA, restart, unzip a FRESH copy of DFL and try again. DFL has CUDA already.
~ Fake it till you make it ~
You are not allowed to view links. Register or Login to view.

Forum Jump:

Users browsing this thread: 1 Guest(s)