My new laptop:
Acer
Intel(R) Core(TM) i7-9750H CPU @ 2.60GhZ 2.59GHZ
(RAM): 32GB (31,8 GB)
64-bits, x64-processor
Windows10
512 GB SSD + 1 TB HDD
NVIDEA GeForce GTX 1650
Downloaded both DeepFaceLab_CUDA_9.2_SSE_build_11_12_2019 and DeepFaceLab_CUDA_10.1_AVX_build_11_14_2019 on my D: drive
First i deinstalled cuda 10.1.
Then like you said get the latest CPU drivers by opening GeForce Experience created a NVIDEA account and installed and then upgraded the latest version of the GeForce Game Ready Driver.
Is this ok?
Then tried again to run DeepFaceLab_CUDA_10.1_AVX_build_11_14_2019. Then at step 6 i got this error:
Running trainer.
Loading model...
Model first run.
Enable autobackup? (y/n ?:help skip:n) : y
Write preview history? (y/n ?:help skip:n) : y
Choose image for the preview history? (y/n skip:n) : y
Target iteration (skip:unlimited/default) :
0
Batch_size (?:help skip:0) : ?
Larger batch size is better for NN's generalization, but it can cause Out of Memory error. Tune this value for your videocard manually.
Batch_size (?:help skip:0) : 4
Feed faces to network sorted by yaw? (y/n ?:help skip:n) :
n
Flip faces randomly? (y/n ?:help skip:y) :
y
Src face scale modifier % ( -30...30, ?:help skip:0) :
0
Resolution ( 64-256 ?:help skip:128) :
128
Half or Full face? (h/f, ?:help skip:f) :
f
Learn mask? (y/n, ?:help skip:y ) :
y
Optimizer mode? ( 1,2,3 ?:help skip:1) :
1
AE architecture (df, liae ?:help skip:df) :
df
AutoEncoder dims (32-1024 ?:help skip:512) :
512
Encoder dims per channel (21-85 ?:help skip:42) :
42
Decoder dims per channel (10-85 ?:help skip:21) :
21
Use CA weights? (y/n, ?:help skip:n ) :
n
Use pixel loss? (y/n, ?:help skip:n ) :
n
Face style power ( 0.0 .. 100.0 ?:help skip:0.00) : 1
Background style power ( 0.0 .. 100.0 ?:help skip:0.00) : 1
Color transfer mode apply to src faceset. ( none/rct/lct/mkl/idt/sot, ?:help skip:none) : rct
Enable gradient clipping? (y/n, ?:help skip:n) :
n
Pretrain the model? (y/n, ?:help skip:n) : y
Using TensorFlow backend.
Loading: 100%|##################################################################| 24711/24711 [01:07<00:00, 366.94it/s]
Choose image for the preview history. [p] - next. [enter] - confirm.
=============== Model Summary ===============
== ==
== Model name: SAE ==
== ==
== Current iteration: 0 ==
== ==
==------------- Model Options -------------==
== ==
== autobackup: True ==
== write_preview_history: True ==
== sort_by_yaw: False ==
== random_flip: True ==
== resolution: 128 ==
== face_type: f ==
== learn_mask: True ==
== optimizer_mode: 1 ==
== archi: df ==
== ae_dims: 512 ==
== e_ch_dims: 42 ==
== d_ch_dims: 21 ==
== ca_weights: False ==
== pixel_loss: False ==
== face_style_power: 1.0 ==
== bg_style_power: 1.0 ==
== ct_mode: rct ==
== clipgrad: False ==
== pretrain: True ==
== batch_size: 4 ==
== ==
==-------------- Running On ---------------==
== ==
== Device index: 0 ==
== Name: GeForce GTX 1650 ==
== VRAM: 4.00GB ==
== ==
=============================================
Starting. Press "Enter" to stop training and save model.
Error: OOM when allocating tensor with shape[64512,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node mul_69}} = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Adam/beta_2/read, Variable_86/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[{{node Mean_11/_1133}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_8203_Mean_11", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Traceback (most recent call last):
File "D:\DeepFaceLab_CUDA_10.1_AVX\_internal\DeepFaceLab\mainscripts\Trainer.py", line 109, in trainerThread
iter, iter_time = model.train_one_iter()
File "D:\DeepFaceLab_CUDA_10.1_AVX\_internal\DeepFaceLab\models\ModelBase.py", line 525, in train_one_iter
losses = self.onTrainOneIter(sample, self.generator_list)
File "D:\DeepFaceLab_CUDA_10.1_AVX\_internal\DeepFaceLab\models\Model_SAE\Model.py", line 509, in onTrainOneIter
src_loss, dst_loss, = self.src_dst_train (feed)
File "D:\DeepFaceLab_CUDA_10.1_AVX\_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py", line 2715, in __call__
return self._call(inputs)
File "D:\DeepFaceLab_CUDA_10.1_AVX\_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py", line 2675, in _call
fetched = self._callable_fn(*array_vals)
File "D:\DeepFaceLab_CUDA_10.1_AVX\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1439, in __call__
run_metadata_ptr)
File "D:\DeepFaceLab_CUDA_10.1_AVX\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 528, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[64512,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node mul_69}} = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Adam/beta_2/read, Variable_86/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[{{node Mean_11/_1133}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_8203_Mean_11", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Then i tried to run DeepFaceLab_CUDA_9.2_SSE_build_11_12_2019
And again this error appeared:
Running trainer.
Loading model...
Model first run.
Enable autobackup? (y/n ?:help skip:n) : y
Write preview history? (y/n ?:help skip:n) : y
Choose image for the preview history? (y/n skip:n) :
n
Target iteration (skip:unlimited/default) :
0
Batch_size (?:help skip:0) : 3
Feed faces to network sorted by yaw? (y/n ?:help skip:n) :
n
Flip faces randomly? (y/n ?:help skip:y) :
y
Src face scale modifier % ( -30...30, ?:help skip:0) :
0
Resolution ( 64-256 ?:help skip:128) :
128
Half or Full face? (h/f, ?:help skip:f) :
f
Learn mask? (y/n, ?:help skip:y ) :
y
Optimizer mode? ( 1,2,3 ?:help skip:1) :
1
AE architecture (df, liae ?:help skip:df) :
df
AutoEncoder dims (32-1024 ?:help skip:512) :
512
Encoder dims per channel (21-85 ?:help skip:42) :
42
Decoder dims per channel (10-85 ?:help skip:21) :
21
Use CA weights? (y/n, ?:help skip:n ) :
n
Use pixel loss? (y/n, ?:help skip:n ) :
n
Face style power ( 0.0 .. 100.0 ?:help skip:0.00) : 1
Background style power ( 0.0 .. 100.0 ?:help skip:0.00) : 1
Color transfer mode apply to src faceset. ( none/rct/lct/mkl/idt, ?:help skip:none) : lct
Enable gradient clipping? (y/n, ?:help skip:n) :
n
Pretrain the model? (y/n, ?:help skip:n) :
n
Using TensorFlow backend.
Loading: 100%|#######################################################################| 632/632 [00:09<00:00, 68.57it/s]
Loading: 100%|#####################################################################| 2175/2175 [00:37<00:00, 57.92it/s]
=============== Model Summary ===============
== ==
== Model name: SAE ==
== ==
== Current iteration: 0 ==
== ==
==------------- Model Options -------------==
== ==
== autobackup: True ==
== write_preview_history: True ==
== sort_by_yaw: False ==
== random_flip: True ==
== resolution: 128 ==
== face_type: f ==
== learn_mask: True ==
== optimizer_mode: 1 ==
== archi: df ==
== ae_dims: 512 ==
== e_ch_dims: 42 ==
== d_ch_dims: 21 ==
== ca_weights: False ==
== pixel_loss: False ==
== face_style_power: 1.0 ==
== bg_style_power: 1.0 ==
== ct_mode: lct ==
== clipgrad: False ==
== batch_size: 3 ==
== ==
==-------------- Running On ---------------==
== ==
== Device index: 0 ==
== Name: GeForce GTX 1650 ==
== VRAM: 4.00GB ==
== ==
=============================================
Starting. Press "Enter" to stop training and save model.
Error: OOM when allocating tensor with shape[64512,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node mul_69}} = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Adam/beta_2/read, Variable_86/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[{{node Mean_11/_1133}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_8203_Mean_11", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Traceback (most recent call last):
File "D:\DeepFaceLab_CUDA_9.2_SSE\_internal\DeepFaceLab\mainscripts\Trainer.py", line 109, in trainerThread
iter, iter_time = model.train_one_iter()
File "D:\DeepFaceLab_CUDA_9.2_SSE\_internal\DeepFaceLab\models\ModelBase.py", line 525, in train_one_iter
losses = self.onTrainOneIter(sample, self.generator_list)
File "D:\DeepFaceLab_CUDA_9.2_SSE\_internal\DeepFaceLab\models\Model_SAE\Model.py", line 509, in onTrainOneIter
src_loss, dst_loss, = self.src_dst_train (feed)
File "D:\DeepFaceLab_CUDA_9.2_SSE\_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py", line 2715, in __call__
return self._call(inputs)
File "D:\DeepFaceLab_CUDA_9.2_SSE\_internal\python-3.6.8\lib\site-packages\keras\backend\tensorflow_backend.py", line 2675, in _call
fetched = self._callable_fn(*array_vals)
File "D:\DeepFaceLab_CUDA_9.2_SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1439, in __call__
run_metadata_ptr)
File "D:\DeepFaceLab_CUDA_9.2_SSE\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 528, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[64512,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node mul_69}} = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](Adam/beta_2/read, Variable_86/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[{{node Mean_11/_1133}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_8203_Mean_11", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Does my new laptop need anything more?
Are my settings ok?
What am i doing wrong?
Can anyone help me please?