MrDeepFakes Forums
  • New and improved dark forum theme!
  • Guests can now comment on videos on the tube.
   
  •  Previous
  • 1
  • 2
  • 3(current)
  • 4
  • 5
  • ...
  • 48
  • Next 
[DISCUSSION] DeepFaceLab 2.0 - Discussion, tips, suggestions
#21
First tests look very promising, but I'm having problems with the video codec.
Is there a way to have the h.265 to work on premiere, AE, vlc and so on?
I can just open the result with media player, and can't convert it from there Sad
#22
Hi all .
I 've been searching the forum left and right but can't really find the answer for this.
i've been using DFL for a few month. was running with a gtx 970 / 4Gig before and never had a tensor flow  / failed to allocate memory problem.
But now that i changed to a " asus strix 2070Super a8g " i get that error  failed to allocate 2.16 Gig.
DFL01 or 02 its the same. i tried DDU to do a fresh install the same.
I launched GTA 5 trying to  max the usage of vram (8 gig) and i saw in the asus gpu tweak software that it did use all the vram. I didnt run it for a long time,  it was just to see if i could reach those 8 gig of vram.
So can someone inlight me. 
is it a : 
-Faulty Graphic card? 
- not important as long as i can train the model even if a cant select better resolution or options.
- something else.

Thank you. 

My spec :
windows 10 64b
I7 - 5820k 3.3ghz
32 gig of ram
and of corse rtx 2070 super
#23
(01-26-2020, 04:15 PM)Takezo Wrote: You are not allowed to view links. Register or Login to view.Hi all .
I 've been searching the forum left and right but can't really find the answer for this.
i've been using DFL for a few month. was running with a gtx 970 / 4Gig before and never had a tensor flow  / failed to allocate memory problem.
But now that i changed to a " asus strix 2070Super a8g " i get that error  failed to allocate 2.16 Gig.
DFL01 or 02 its the same. i tried DDU to do a fresh install the same.
I launched GTA 5 trying to  max the usage of vram (8 gig) and i saw in the asus gpu tweak software that it did use all the vram. I didnt run it for a long time,  it was just to see if i could reach those 8 gig of vram.
So can someone inlight me. 
is it a : 
-Faulty Graphic card? 
- not important as long as i can train the model even if a cant select better resolution or options.
- something else.

Thank you. 

My spec :
windows 10 64b
I7 - 5820k 3.3ghz
32 gig of ram
and of corse rtx 2070 super

You are not allowed to view links. Register or Login to view.
As this test says. It could be time to upgrade that processor. I know dfl is mainly a gpu heavy software. But when windows steal some of that vram, and you don't have integrated graphics, that could be a lot. The failed to allocate memory might be because of all that.
#24
(01-26-2020, 04:58 PM)Groggy4 Wrote: You are not allowed to view links. Register or Login to view.
(01-26-2020, 04:15 PM)Takezo Wrote: You are not allowed to view links. Register or Login to view.Hi all .
I 've been searching the forum left and right but can't really find the answer for this.
i've been using DFL for a few month. was running with a gtx 970 / 4Gig before and never had a tensor flow  / failed to allocate memory problem.
But now that i changed to a " asus strix 2070Super a8g " i get that error  failed to allocate 2.16 Gig.
DFL01 or 02 its the same. i tried DDU to do a fresh install the same.
I launched GTA 5 trying to  max the usage of vram (8 gig) and i saw in the asus gpu tweak software that it did use all the vram. I didnt run it for a long time,  it was just to see if i could reach those 8 gig of vram.
So can someone inlight me. 
is it a : 
-Faulty Graphic card? 
- not important as long as i can train the model even if a cant select better resolution or options.
- something else.

Thank you. 

My spec :
windows 10 64b
I7 - 5820k 3.3ghz
32 gig of ram
and of corse rtx 2070 super

You are not allowed to view links. Register or Login to view.
As this test says. It could be time to upgrade that processor. I know dfl is mainly a gpu heavy software. But when windows steal some of that vram, and you don't have integrated graphics, that could be a lot. The failed to allocate memory might be because of all that.

Wow thank you. i never could have imagine that the Cpu might be the problem. since it was a good one at the time i bought it ^^ .
Thank you again. Time to look at my wallet see what can be done. 
And thank for the link very usefull.
#25
@Takezo Yeah, I'm not saying the problem is 100% that. But when the technology in CPU outdate new GPU by 5 years. Some bottleneck issues is bound to happen.
#26
better post the exact error log in here befor buy something new....
#27
Running trainer.

Choose one of saved models, or enter a name to create a new model.
[r] : rename
[d] : delete

[0] : momo - latest
 :
0
Loading momo_SAEHD model...

Choose one or several GPU idxs (separated by comma).

[CPU] : CPU
  [0] : GeForce RTX 2070 SUPER

[0] Which GPU indexes to choose? :
0

Initializing models: 100%|#######################| 7/7 [00:04<00:00,  1.52it/s]
Loading samples: 100%|####################| 4338/4338 [00:06<00:00, 644.78it/s]
Loading samples: 100%|####################| 4446/4446 [00:06<00:00, 701.60it/s]
2020-01-26 13:54:52.465070: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:54:52.469575: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:54:52.510821: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:54:52.517593: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:54:52.556598: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:54:52.561707: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:54:52.588447: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:54:52.593617: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:54:52.605749: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:54:52.616764: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:54:52.754514: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:54:52.770399: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:54:52.927200: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:54:52.932553: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
================== Model Summary ==================
==                                               ==
==            Model name: momo_SAEHD             ==
==                                               ==
==     Current iteration: 61589                  ==
==                                               ==
==---------------- Model Options ----------------==
==                                               ==
==            autobackup: False                  ==
== write_preview_history: False                  ==
==           target_iter: 0                      ==
==           random_flip: True                   ==
==            resolution: 64                     ==
==             face_type: f                      ==
==     models_opt_on_gpu: False                  ==
==                 archi: dfhd                   ==
==               ae_dims: 256                    ==
==                e_dims: 64                     ==
==                d_dims: 64                     ==
==           d_mask_dims: 22                     ==
==            learn_mask: False                  ==
==            lr_dropout: True                   ==
==           random_warp: False                  ==
==    true_face_training: True                   ==
==      face_style_power: 0.1                    ==
==        bg_style_power: 0.1                    ==
==               ct_mode: rct                    ==
==              clipgrad: False                  ==
==              pretrain: False                  ==
==            batch_size: 20                     ==
==                                               ==
==----------------- Running On ------------------==
==                                               ==
==          Device index: 0                      ==
==                  Name: GeForce RTX 2070 SUPER ==
==                  VRAM: 8.00GB                 ==
==                                               ==
===================================================
Starting. Press "Enter" to stop training and save model.
2020-01-26 13:55:06.711916: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:06.717314: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:06.868834: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:06.874226: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:06.922721: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:06.927965: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:07.015578: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:07.020742: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:07.066575: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:07.071787: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:07.136643: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:07.141836: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:08.020504: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:08.026375: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:08.059312: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:08.065282: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:08.096824: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:08.102420: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:08.692536: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:08.699070: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:08.728492: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:08.734032: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:08.760446: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:08.766076: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:09.189555: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 13:55:09.195273: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.16G (2318294528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
[14:10:02][#062454][0932ms][0.1679][0.1683]
[14:25:10][#063350][1016ms][0.1742][0.1761]
[14:40:19][#064256][1024ms][0.1746][0.1769]
Done.
Appuyez sur une touche pour continuer...


the same thing for bigger resolution and tested it with a batch size of 2 . always the same error.
thank you for the help.
#28
increase ur windows swap. maybe u have to less RAM
models_opt_on_gpu: False <--- set on True
why res 64?
remove that: == face_style_power: 0.1 ==
== bg_style_power: 0.1

maybe u better read the tutorials...
#29
(01-26-2020, 05:46 PM)aXu Wrote: You are not allowed to view links. Register or Login to view.increase ur windows swap. maybe u have to less RAM
models_opt_on_gpu: False <--- set on True
why res 64?
remove that: ==      face_style_power: 0.1                    ==
==        bg_style_power: 0.1

maybe u better read the tutorials...

For the res of 64 it was because i wanted to test the DFL 2 on a project i was already doing. So i didnt want to spend to much time.
The face style and bg style I get the same error with or without it. 
For the tutorial i did read them when praticing with the old gtx970. and never had this problem.

here my log from the last DFL 01 version.

=====================================

Running trainer.

Loading model...
Press enter in 2 seconds to override model settings.
Enable autobackup? (y/n ?:help skip:y) :
y
Write preview history? (y/n ?:help skip:y) :
y
Choose image for the preview history? (y/n skip:n) :
n
Target iteration (skip:unlimited/default) :
0
Batch_size (?:help skip:7) :
7
Flip faces randomly? (y/n ?:help skip:y) :
y
Learn mask? (y/n, ?:help skip:y ) :
y
Optimizer mode? ( 1,2,3 ?:help skip:2) :
2
Use learning rate dropout? (y/n, ?:help skip:y ) :
y
Enable random warp of samples? ( y/n, ?:help skip:n) :
n
Enable 'true face' training? (y/n, ?:help skip:n) :
n
Face style power ( 0.0 .. 100.0 ?:help skip:0.30) : 0
Background style power ( 0.0 .. 100.0 ?:help skip:0.20) : 0
Color transfer mode apply to src faceset. ( none/rct/lct/mkl/idt/sot, ?:help skip:idt) :
idt
Enable gradient clipping? (y/n, ?:help skip:y) :
y
Using TensorFlow backend.
Loading: 100%|#############################| 4338/4338 [01:43<00:00, 41.99it/s]
Loading: 100%|#############################| 4482/4482 [00:56<00:00, 79.89it/s]
================== Model Summary ==================
==                                               ==
==            Model name: SAEHD                  ==
==                                               ==
==     Current iteration: 195456                 ==
==                                               ==
==---------------- Model Options ----------------==
==                                               ==
==            autobackup: True                   ==
== write_preview_history: True                   ==
==           random_flip: True                   ==
==            resolution: 128                    ==
==             face_type: f                      ==
==            learn_mask: True                   ==
==        optimizer_mode: 2                      ==
==                 archi: df                     ==
==               ae_dims: 256                    ==
==            ed_ch_dims: 21                     ==
==            lr_dropout: True                   ==
==           random_warp: False                  ==
==    true_face_training: False                  ==
==      face_style_power: 0.0                    ==
==        bg_style_power: 0.0                    ==
==               ct_mode: idt                    ==
==              clipgrad: True                   ==
==            batch_size: 7                      ==
==                                               ==
==----------------- Running On ------------------==
==                                               ==
==          Device index: 0                      ==
==                  Name: GeForce RTX 2070 SUPER ==
==                  VRAM: 8.00GB                 ==
==                                               ==
===================================================
Starting. Press "Enter" to stop training and save model.
2020-01-26 16:02:22.631023: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.15G (2305711616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
[16:02:48][#195468][1015ms][0.1816][0.1107]

=============================================

Im gonna try the setup you told me with DFL 02
and here are the result.

================================
Running trainer.

Choose one of saved models, or enter a name to create a new model.
[r] : rename
[d] : delete

[0] : momo - latest
 :
0
Loading momo_SAEHD model...

Choose one or several GPU idxs (separated by comma).

[CPU] : CPU
  [0] : GeForce RTX 2070 SUPER

[0] Which GPU indexes to choose? :
0

Press enter in 2 seconds to override model settings.
[n] Enable autobackup ( y/n ?:help ) :
n
[n] Write preview history ( y/n ?:help ) :
n
[0] Target iteration :
0
[y] Flip faces randomly ( y/n ?:help ) :
y
[20] Batch_size ( ?:help ) :
20
[n] Place models and optimizer on GPU ( y/n ?:help ) : y
[n] Learn mask ( y/n ?:help ) :
n
[y] Use learning rate dropout ( y/n ?:help ) :
y
[n] Enable random warp of samples ( y/n ?:help ) :
n
[y] Enable 'true face' training ( y/n ?:help ) :
y
[0.1] Face style power ( 0.0..100.0 ?:help ) : 0
0.0
[0.1] Background style power ( 0.0..100.0 ?:help ) : 0
0.0
[rct] Color transfer for src faceset ( none/rct/lct/mkl/idt/sot ?:help ) :
rct
[n] Enable gradient clipping ( y/n ?:help ) :
n
[n] Enable pretraining mode ( y/n ?:help ) :
n
Initializing models: 100%|#######################| 7/7 [00:04<00:00,  1.72it/s]
Loading samples: 100%|####################| 4338/4338 [00:07<00:00, 613.79it/s]
Loading samples: 100%|####################| 4446/4446 [00:06<00:00, 680.97it/s]
2020-01-26 16:06:26.794535: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:26.799414: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:26.850042: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:26.854971: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:26.929871: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:26.934530: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:26.996889: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:27.001715: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
================== Model Summary ==================
==                                               ==
==            Model name: momo_SAEHD             ==
==                                               ==
==     Current iteration: 65113                  ==
==                                               ==
==---------------- Model Options ----------------==
==                                               ==
==            autobackup: False                  ==
== write_preview_history: False                  ==
==           target_iter: 0                      ==
==           random_flip: True                   ==
==            resolution: 64                     ==
==             face_type: f                      ==
==     models_opt_on_gpu: True                   ==
==                 archi: dfhd                   ==
==               ae_dims: 256                    ==
==                e_dims: 64                     ==
==                d_dims: 64                     ==
==           d_mask_dims: 22                     ==
==            learn_mask: False                  ==
==            lr_dropout: True                   ==
==           random_warp: False                  ==
==    true_face_training: True                   ==
==      face_style_power: 0.0                    ==
==        bg_style_power: 0.0                    ==
==               ct_mode: rct                    ==
==              clipgrad: False                  ==
==              pretrain: False                  ==
==            batch_size: 20                     ==
==                                               ==
==----------------- Running On ------------------==
==                                               ==
==          Device index: 0                      ==
==                  Name: GeForce RTX 2070 SUPER ==
==                  VRAM: 8.00GB                 ==
==                                               ==
===================================================
Starting. Press "Enter" to stop training and save model.
2020-01-26 16:06:38.840453: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:38.845304: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:38.978748: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:38.983647: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.041358: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.046077: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.050840: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.055454: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.155561: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.160365: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.203428: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.208243: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.222975: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.228667: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.233923: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.238548: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.300094: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.304944: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.332816: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.337528: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.690399: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.695566: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.724767: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.729560: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.786533: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.797972: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.808008: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:39.823273: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:40.279343: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:40.284225: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:40.313330: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:40.318044: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:43.000599: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:44.616973: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:44.621746: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:46.155341: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2020-01-26 16:06:54.034926: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.12G (2280545792 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
[16:07:20][#065180][0539ms][0.1724][0.1876]

================================================================
i 've increase the windows swap but same
thank you
#30
try archi: df instead of dfhd.
and check in taskmanager how much ram it uses and how much ram and how much swap
und my 2080ti i can use just something about 9gb vram of the 11gb...
  •  Previous
  • 1
  • 2
  • 3(current)
  • 4
  • 5
  • ...
  • 48
  • Next 

Forum Jump:

Users browsing this thread: 1 Guest(s)