MrDeepFakes Forums

Some content may not be available to Guests. Consider registering an account to enjoy unrestricted access to guides, support and tools

  • We are looking for community members who are intested in helping out. See our HELP WANTED post.

Error with DeepFaceLab

Baz4k

DF Vagrant
I am pulling my hair out. I get to number 6 and I get errors. I can't figure it out to save my life...I have been searing for 3 days on how to fix this...anyone have any ideas??? I have tried every batch file for 6 and they all do pretty much the same...I have tried installing Python and Cuda...and tried it without them...Any help is appreciated.

Running trainer.

Loading model...

Model first run. Enter model options as default for each run.
Enable autobackup? (y/n ?:help skip:n) : y
Write preview history? (y/n ?:help skip:n) : y
Choose image for the preview history? (y/n skip:n) : n
Target iteration (skip:unlimited/default) :
0
Batch_size (?:help skip:0) : 0
Feed faces to network sorted by yaw? (y/n ?:help skip:n) : y
Flip faces randomly? (y/n ?:help skip:y) : y
Src face scale modifier % ( -30...30, ?:help skip:0) : 0
Use lightweight autoencoder? (y/n, ?:help skip:n) : n
Use pixel loss? (y/n, ?:help skip: n/default ) : n
Using TensorFlow backend.
Loading: 100%|######################################################################| 559/559 [00:00<00:00, 934.12it/s]
Sorting: 100%|#######################################################################| 64/64 [00:00<00:00, 4936.38it/s]
Loading: 100%|##################################################################| 23610/23610 [00:26<00:00, 876.59it/s]
Sorting: 100%|########################################################################| 64/64 [00:00<00:00, 128.42it/s]
===== Model summary =====
== Model name: H128
==
== Current iteration: 0
==
== Model options:
== |== autobackup : True
== |== write_preview_history : True
== |== batch_size : 4
== |== sort_by_yaw : True
== |== random_flip : True
== |== lighter_ae : False
== |== pixel_loss : False
== Running on:
== |== [0 : GeForce GTX 1080]
=========================
Starting. Press "Enter" to stop training and save model.
2019-08-15 12:59:05.704491: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.707619: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.711253: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.715047: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.730485: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.736032: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.738912: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.742181: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.745097: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.750437: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.754749: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:05.758638: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2019-08-15 12:59:09.601209: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
2019-08-15 12:59:09.642917: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
2019-08-15 12:59:09.645225: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
2019-08-15 12:59:09.649229: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
Error: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
         [[{{node model_1/conv2d_1/convolution}} = Conv2D[T=DT_FLOAT, _class=["loc:mad:train...propFilter"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](training/Adam/gradients/model_1/conv2d_1/convolution_grad/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOptimizer, conv2d_1/kernel/read)]]
         [[{{node loss/model_3_loss/Mean_3/_579}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4418_loss/model_3_loss/Mean_3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Traceback (most recent call last):
  File "C:\Users\bizzy\Desktop\DeepFaceLabCUDA9.2SSE\_internal\DeepFaceLab\mainscripts\Trainer.py", line 107, in trainerThread
    iter, iter_time = model.train_one_iter()
  File "C:\Users\bizzy\Desktop\DeepFaceLabCUDA9.2SSE\_internal\DeepFaceLab\models\ModelBase.py", line 472, in train_one_iter
    losses = self.onTrainOneIter(sample, self.generator_list)
  File "C:\Users\bizzy\Desktop\DeepFaceLabCUDA9.2SSE\_internal\DeepFaceLab\models\Model_H128\Model.py", line 88, in onTrainOneIter
    total, loss_src_bgr, loss_src_mask, loss_dst_bgr, loss_dst_mask = self.ae.train_on_batch( [warped_src, target_src_mask, warped_dst, target_dst_mask], [target_src, target_src_mask, target_dst, target_dst_mask] )
  File "C:\Users\bizzy\Desktop\DeepFaceLabCUDA9.2SSE\_internal\python-3.6.8\lib\site-packages\keras\engine\training.py", line 1217, in train_on_batch
    outputs = self.train_function(ins)
  File "C:\Users\bizzy\Desktop\DeepFaceLabCUDA9.2SSE\_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py", line 2715, in __call__
    return self._call(inputs)
  File "C:\Users\bizzy\Desktop\DeepFaceLabCUDA9.2SSE\_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py", line 2675, in _call
    fetched = self._callable_fn(*array_vals)
  File "C:\Users\bizzy\Desktop\DeepFaceLabCUDA9.2SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1439, in __call__
    run_metadata_ptr)
  File "C:\Users\bizzy\Desktop\DeepFaceLabCUDA9.2SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 528, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
         [[{{node model_1/conv2d_1/convolution}} = Conv2D[T=DT_FLOAT, _class=["loc:mad:train...propFilter"], data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](training/Adam/gradients/model_1/conv2d_1/convolution_grad/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOptimizer, conv2d_1/kernel/read)]]
         [[{{node loss/model_3_loss/Mean_3/_579}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4418_loss/model_3_loss/Mean_3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Done.
Press any key to continue . . .
 

dpfks

DF Enthusiast
Staff member
Administrator
Verified Video Creator
Baz4k said:
I just tried reinstalling Cuda 9.2 and clearing the workspace and it did the same thing.

Uninstall CUDA, restart, unzip a FRESH copy of DFL and try again. DFL has CUDA already.
 
Top