MrDeepFakes Forums

Some content may not be available to Guests. Consider registering an account to enjoy unrestricted access to guides, support and tools

  • We are looking for community members who are intested in helping out. See our HELP WANTED post.

Crashes as soon as training starts on 1080 ti

yushy97

DF Vagrant
Hello. 

I've just started trying to make deepfakes with DeepFaceLab on my 1080ti and I've been getting the same problem every time I try to train the program. I go through the process of creating them (extracting the frames, aligning, manually removing frames etc...), but when I get to the process of training, my PC crashes and restarts.

This was a problem with FakeApp as well when I previously tried using that. Is this a known issue and are there any troubleshooting steps to fix this? Is there a log somewhere that might point towards the problem?
 

iperov

DF Enthusiast
Developer
change thermal grease on CPU/GPU.
decrease DRAM frequency
decrease CPU/GPU frequencies.
 

frosty3907

DF Admirer
Verified Video Creator
yushy97 said:
Hello. 

I've just started trying to make deepfakes with DeepFaceLab on my 1080ti and I've been getting the same problem every time I try to train the program. I go through the process of creating them (extracting the frames, aligning, manually removing frames etc...), but when I get to the process of training, my PC crashes and restarts.

This was a problem with FakeApp as well when I previously tried using that. Is this a known issue and are there any troubleshooting steps to fix this? Is there a log somewhere that might point towards the problem?

Yeah that sounds like an issue with your system's stability, try running burn in tests like furmark - install hwinfo etc to see if your temps are getting out of hand. Do you have an adequate power supply?
 

yushy97

DF Vagrant
iperov said:
change thermal grease on CPU/GPU.
decrease DRAM frequency
decrease CPU/GPU frequencies.

I'll try that. Thanks!


frosty3907 said:
yushy97 said:
Hello. 

I've just started trying to make deepfakes with DeepFaceLab on my 1080ti and I've been getting the same problem every time I try to train the program. I go through the process of creating them (extracting the frames, aligning, manually removing frames etc...), but when I get to the process of training, my PC crashes and restarts.

This was a problem with FakeApp as well when I previously tried using that. Is this a known issue and are there any troubleshooting steps to fix this? Is there a log somewhere that might point towards the problem?

Yeah that sounds like an issue with your system's stability, try running burn in tests like furmark - install hwinfo etc to see if your temps are getting out of hand. Do you have an adequate power supply?
I'll ran the Unigine Superposition VR benchmark on the Future setting and it lasted for about a minute before it crashed and restarted, so I think you're right about the system's instability. 
I don't think its a PSU issue though, because I've got a Corsair HX850W so I think it should be alright.
I will install hwinfo and check the temps. 

Thanks for the help!
 

andy_ger

DF Vagrant
Hi yushy,

i had the same problems with my new pc.
A clean reinstall of Windows solved the problem for me.

I think, that some drivers have been damaged.
But a simple reinstall of the Nvidia drivers did not help.
 

Notorious

DF Vagrant
Hi everyone,

I just purchase the 1080 Ti also, when I start to train or convert it got stuck at:

Y:\_internal\bin\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
2019-02-25 21:07:12.296315: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1411] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.62
pciBusID: 0000:01:00.0
totalMemory: 11.00GiB freeMemory: 9.11GiB
2019-02-25 21:07:12.304177: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1490] Adding visible gpu devices: 0


I installed the latest drivers and the latest DFL, so eventually I tried to wait for as long as possible and it finally ran, it took around 5 minutes or more till it eventually run, is that normal? Before this I was using CPU and I didn't have to wait that long.

Thanks
 

Pololo

DF Vagrant
This is happening to me today, it has been following the last update of nvidia, before it did not happen to me.

Last nvidia update: 419.17
 

dpfks

DF Enthusiast
Staff member
Administrator
Verified Video Creator
Notorious said:
Hi everyone,

I just purchase the 1080 Ti also, when I start to train or convert it got stuck at:

Y:\_internal\bin\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
2019-02-25 21:07:12.296315: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1411] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.62
pciBusID: 0000:01:00.0
totalMemory: 11.00GiB freeMemory: 9.11GiB
2019-02-25 21:07:12.304177: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1490] Adding visible gpu devices: 0


I installed the latest drivers and the latest DFL, so eventually I tried to wait for as long as possible and it finally ran, it took around 5 minutes or more till it eventually run, is that normal? Before this I was using CPU and I didn't have to wait that long.

Thanks

Pololo said:
This is happening to me today, it has been following the last update of nvidia, before it did not happen to me.

Last nvidia update: 419.17

The first run usually takes the longest. It usually starts faster after that.
 

Notorious

DF Vagrant
Pololo said:
This is happening to me today, it has been following the last update of nvidia, before it did not happen to me.

Last nvidia update: 419.17

Oh i see, I never try any version below that one you mentioned, hmm

dpfks said:
Notorious said:
Hi everyone,

I just purchase the 1080 Ti also, when I start to train or convert it got stuck at:

Y:\_internal\bin\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
2019-02-25 21:07:12.296315: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1411] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.62
pciBusID: 0000:01:00.0
totalMemory: 11.00GiB freeMemory: 9.11GiB
2019-02-25 21:07:12.304177: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1490] Adding visible gpu devices: 0


I installed the latest drivers and the latest DFL, so eventually I tried to wait for as long as possible and it finally ran, it took around 5 minutes or more till it eventually run, is that normal? Before this I was using CPU and I didn't have to wait that long.

Thanks

Pololo said:
This is happening to me today, it has been following the last update of nvidia, before it did not happen to me.

Last nvidia update: 419.17

The first run usually takes the longest. It usually starts faster after that.

Oh that's alright then, I thought something is wrong with my GPU :p
 
Top