ColeRogan said:
tommo020788 said:
EDIT: Figured it out. Didn't notice there was a vram mem error, and what I was looking at was a collapsed model.
Reduced a couple settings and now its working
I'm using 1080ti graphics card.
Where did you see the error? What settings did you reduce? I'm having a similar if not exact problem. Thanks
I saw the error after it loaded a certain amount in the command window.
When I pulled up a GPU hardware monitor, I could see it was trying to use up just over 11gb of memory when the max my gpu is capable is 11gb (so it was going slightly over its max technical capacity).
The error would happen right after the gpu's memory was maxed out (not like ur typical popup crash error, but rather an error reported in the cmd window).
So all I had to do is change certain settings to reduce gpu usage.
The relevant settings I used were:
Resolution:__________128
Face_type: __________full
Learn_mask:__________y
Archi:_________________df
AE_Dims______________256
E_Ch_Dims:___________The Default Option/hit enter (the default had a lower setting). If you already have, u could try lowering it more.
D_Ch_dims____________Same as e_ch_dims.
Multi-scale_decoder:__y
Ca_weights:___________y
Face_style_power:____10 (u could also set this to 0 if u still have issues, but don't if u can manage not to).
Bg_style_power:______10 (see style power note above)
The Two settings that reduced gpu mem usage the most were: E_Ch_Dims and D_Ch_dims. I believe the trade off of lowering those two settings is that it would process training slower, but I'm not certain.
P.s. ^These two settings mentioned just above are abbreviations of two setting names, You'll notice which settings fit those abbreviations as you go through all the settings in the SAE training window, I just can't remember the full name of those settings off the top of my head XD
Hope that helps in some way.