MrDeepFakes Forums
  • New and improved dark forum theme!
  • Guests can now comment on videos on the tube.
   
TrialityDeepFaceLab won't train error code provided...
#1
I started off 2 days ago using the DeepFaceLabOpenCLSSE_build_09_07_2019.exe and it ran fine albeit very slowly off the HD 2000 card  onboard graphics.

So I installed a 2GB 750 ti card  I had lying around and installed  DeepFaceLabCUDA9.2SSE_build_09_07_2019.exe and everything works fine until I go to start training and it never starts. I tried deleting the folder entirely and reinstalling it fresh, updated graphics card drivers, rebooted, deleted all Cuda and python installed outside of DeepFaceLab and still no dice. Any ideas?? Posted what it says on both h64 and h128.

Win 10
I5 7th Gen
12 GB ram
750 ti

/!\

Starting. Press "Enter" to stop training and save model.

Error: OOM when allocating tensor with shape[3,3,512,2048] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

         [[node training/Adam/Variable_60/Assign (defined at C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py:402)  = Assign[T=DT_FLOAT, _grappler_relax_allocator_constraints=true, use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](training/Adam/Variable_60, training/Adam/zeros_14)]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.





Caused by op 'training/Adam/Variable_60/Assign', defined at:

  File "threading.py", line 884, in _bootstrap

  File "threading.py", line 916, in _bootstrap_inner

  File "threading.py", line 864, in run

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\DeepFaceLab\mainscripts\Trainer.py", line 108, in trainerThread

    iter, iter_time = model.train_one_iter()

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\DeepFaceLab\models\ModelBase.py", line 492, in train_one_iter

    losses = self.onTrainOneIter(sample, self.generator_list)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\DeepFaceLab\models\Model_H64\Model.py", line 89, in onTrainOneIter

    total, loss_src_bgr, loss_src_mask, loss_dst_bgr, loss_dst_mask = self.ae.train_on_batch( [warped_src, target_src_full_mask, warped_dst, target_dst_full_mask], [target_src, target_src_full_mask, target_dst, target_dst_full_mask] )

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\keras\engine\training.py", line 1216, in train_on_batch

    self._make_train_function()

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\keras\engine\training.py", line 509, in _make_train_function

    loss=self.total_loss)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\DeepFaceLab\nnlib\nnlib.py", line 638, in get_updates

    vs = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\DeepFaceLab\nnlib\nnlib.py", line 638, in <listcomp>

    vs = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py", line 704, in zeros

    return variable(v, dtype=dtype, name=name)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py", line 402, in variable

    v = tf.Variable(value, dtype=tf.as_dtype(dtype), name=name)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 183, in __call__

    return cls._variable_v1_call(*args, **kwargs)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 146, in _variable_v1_call

    aggregation=aggregation)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 125, in <lambda>

    previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 2444, in default_variable_creator

    expected_shape=expected_shape, import_scope=import_scope)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 187, in __call__

    return super(VariableMetaclass, cls).__call__(*args, **kwargs)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 1329, in __init__

    constraint=constraint)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 1481, in _init_from_args

    validate_shape=validate_shape).op

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\state_ops.py", line 221, in assign

    validate_shape=validate_shape)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\gen_state_ops.py", line 61, in assign

    use_locking=use_locking, name=name)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper

    op_def=op_def)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\util\deprecation.py", line 488, in new_func

    return func(*args, **kwargs)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 3274, in create_op

    op_def=op_def)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 1770, in __init__

    self._traceback = tf_stack.extract_stack()



ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[3,3,512,2048] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

         [[node training/Adam/Variable_60/Assign (defined at C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py:402)  = Assign[T=DT_FLOAT, _grappler_relax_allocator_constraints=true, use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](training/Adam/Variable_60, training/Adam/zeros_14)]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.





Traceback (most recent call last):

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call

    return fn(*args)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn

    options, feed_dict, fetch_list, target_list, run_metadata)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun

    run_metadata)

tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3,3,512,2048] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

         [[{{node training/Adam/Variable_60/Assign}} = Assign[T=DT_FLOAT, _grappler_relax_allocator_constraints=true, use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](training/Adam/Variable_60, training/Adam/zeros_14)]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.





During handling of the above exception, another exception occurred:



Traceback (most recent call last):

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\DeepFaceLab\mainscripts\Trainer.py", line 108, in trainerThread

    iter, iter_time = model.train_one_iter()

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\DeepFaceLab\models\ModelBase.py", line 492, in train_one_iter

    losses = self.onTrainOneIter(sample, self.generator_list)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\DeepFaceLab\models\Model_H64\Model.py", line 89, in onTrainOneIter

    total, loss_src_bgr, loss_src_mask, loss_dst_bgr, loss_dst_mask = self.ae.train_on_batch( [warped_src, target_src_full_mask, warped_dst, target_dst_full_mask], [target_src, target_src_full_mask, target_dst, target_dst_full_mask] )

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\keras\engine\training.py", line 1217, in train_on_batch

    outputs = self.train_function(ins)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py", line 2697, in __call__

    if hasattr(get_session(), '_make_callable_from_options'):

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py", line 206, in get_session

    session.run(tf.variables_initializer(uninitialized_vars))

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 929, in run

    run_metadata_ptr)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run

    feed_dict_tensor, options, run_metadata)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run

    run_metadata)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call

    raise type(e)(node_def, op, message)

tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3,3,512,2048] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

         [[node training/Adam/Variable_60/Assign (defined at C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py:402)  = Assign[T=DT_FLOAT, _grappler_relax_allocator_constraints=true, use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](training/Adam/Variable_60, training/Adam/zeros_14)]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.





Caused by op 'training/Adam/Variable_60/Assign', defined at:

  File "threading.py", line 884, in _bootstrap

  File "threading.py", line 916, in _bootstrap_inner

  File "threading.py", line 864, in run

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\DeepFaceLab\mainscripts\Trainer.py", line 108, in trainerThread

    iter, iter_time = model.train_one_iter()

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\DeepFaceLab\models\ModelBase.py", line 492, in train_one_iter

    losses = self.onTrainOneIter(sample, self.generator_list)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\DeepFaceLab\models\Model_H64\Model.py", line 89, in onTrainOneIter

    total, loss_src_bgr, loss_src_mask, loss_dst_bgr, loss_dst_mask = self.ae.train_on_batch( [warped_src, target_src_full_mask, warped_dst, target_dst_full_mask], [target_src, target_src_full_mask, target_dst, target_dst_full_mask] )

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\keras\engine\training.py", line 1216, in train_on_batch

    self._make_train_function()

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\keras\engine\training.py", line 509, in _make_train_function

    loss=self.total_loss)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\DeepFaceLab\nnlib\nnlib.py", line 638, in get_updates

    vs = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\DeepFaceLab\nnlib\nnlib.py", line 638, in <listcomp>

    vs = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py", line 704, in zeros

    return variable(v, dtype=dtype, name=name)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py", line 402, in variable

    v = tf.Variable(value, dtype=tf.as_dtype(dtype), name=name)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 183, in __call__

    return cls._variable_v1_call(*args, **kwargs)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 146, in _variable_v1_call

    aggregation=aggregation)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 125, in <lambda>

    previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 2444, in default_variable_creator

    expected_shape=expected_shape, import_scope=import_scope)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 187, in __call__

    return super(VariableMetaclass, cls).__call__(*args, **kwargs)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 1329, in __init__

    constraint=constraint)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\variables.py", line 1481, in _init_from_args

    validate_shape=validate_shape).op

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\state_ops.py", line 221, in assign

    validate_shape=validate_shape)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\gen_state_ops.py", line 61, in assign

    use_locking=use_locking, name=name)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper

    op_def=op_def)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\util\deprecation.py", line 488, in new_func

    return func(*args, **kwargs)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 3274, in create_op

    op_def=op_def)

  File "C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 1770, in __init__

    self._traceback = tf_stack.extract_stack()



ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[3,3,512,2048] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

         [[node training/Adam/Variable_60/Assign (defined at C:\Users\pc\Desktop\DeepFaceLabCUDA10.1SSE_build_09_07_2019\DeepFaceLabCUDA10.1SSE\_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py:402)  = Assign[T=DT_FLOAT, _grappler_relax_allocator_constraints=true, use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](training/Adam/Variable_60, training/Adam/zeros_14)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.



Forum Jump:

Users browsing this thread: 1 Guest(s)