MrDeepFakes Forums
  • New and improved dark forum theme!
  • Guests can now comment on videos on the tube.
   
EndalusModel Backup batch files
#1
Hi all. I'm paranoid about losing hours of progress on models due to model crashes or the VM I use to train on going down for maintenance unexpectedly during a save cycle, so I wrote a couple batch files to help with backing up models so that I can roll back if something goes bad with the current active model. 

I'm still learning python, so I don't currently know enough to try to edit DFL code directly, but these do alright in the meantime. You can copy this code and save it as a batch file in your workspace directory.

This first one is run manually and will copy your current model to a timestamped backup directory.
Code:
@echo off
set CUR_YYYY=%date:~10,4%
set CUR_MM=%date:~4,2%
set CUR_DD=%date:~7,2%
set CUR_HH=%time:~0,2%
if %CUR_HH% lss 10 (set CUR_HH=0%time:~1,1%)

set CUR_NN=%time:~3,2%
set CUR_SS=%time:~6,2%
set CUR_MS=%time:~9,2%

set SUBFILENAME=Backup-%CUR_YYYY%.%CUR_MM%.%CUR_DD%-%CUR_HH%.%CUR_NN%.%CUR_SS%
mkdir %SUBFILENAME%

xcopy model %SUBFILENAME% /s /e


This second one will backup your model to a timestamped directory every 15 minutes until you close the command window

Code:
:LOOP

set CUR_YYYY=%date:~10,4%
set CUR_MM=%date:~4,2%
set CUR_DD=%date:~7,2%
set CUR_HH=%time:~0,2%
if %CUR_HH% lss 10 (set CUR_HH=0%time:~1,1%)

set CUR_NN=%time:~3,2%
set CUR_SS=%time:~6,2%
set CUR_MS=%time:~9,2%

set SUBFILENAME=Backup-%CUR_YYYY%.%CUR_MM%.%CUR_DD%-%CUR_HH%.%CUR_NN%.%CUR_SS%
mkdir %SUBFILENAME%

xcopy model %SUBFILENAME% /s /e

timeout 900

if not ErrorLevel 1 goto :LOOP

You can also just use windows Task Scheduler to run the manual batch file as a recurring task, but I found that to be kind of flaky, so I just wrote a looping batch file.

Hope this is helpful
#2
This is going to be great help for some people! great work!
#3
Stickied since a lot of users should be using this.

How to use:
1) take code from post 1 and paste into notepad
2) save as name eg: backup.bat - must save as .bat file
3) copy file into /workspace folder and run it while you're training a model
~ Fake it till you make it ~
You are not allowed to view links. Register or Login to view.
#4
One edge case is where the .bat file is copying the model when DFL kicks off a model save. Training will stop as DFL can't rename / delete the model files while they're open by xcopy for the model backup.

Not likely to happen as they're both scheduled for 15min intervals so there shouldn't be too much 'drift' between the two time slices.

I still prefer to have DFL itself do model backup copies, as the above will never happen, plus if training stop for other reasons (oom, etc), no further model backups will happen either.
#5
yayyyy, I got a pinned post
(04-16-2019, 10:49 PM)titan_rw Wrote: You are not allowed to view links. Register or Login to view.One edge case is where the .bat file is copying the model when DFL kicks off a model save.  Training will stop as DFL can't rename / delete the model files while they're open by xcopy for the model backup.

Not likely to happen as they're both scheduled for 15min intervals so there shouldn't be too much 'drift' between the two time slices.

I still prefer to have DFL itself do model backup copies, as the above will never happen, plus if training stop for other reasons (oom, etc), no further model backups will happen either.

I remember someone posted some python code that would run a backup whenever the model saved in the DFL github repo. Unfortunately it was in one of the Free For Talk issue threads that seem to have been deleted now. Unfortunately I don't know enough python to write it from scratch.
In my personal copy I've actually changed 900 (15 minutes) to 910 to help account for collision. On average the backup will lag behind the save by an additional 40 seconds every hour, but you don't have to worry about it for almost a full day. I hit save and reset the backup script when I leave for work and when I get home from dinner and it keeps pretty well synched.
#6
(04-17-2019, 05:03 AM)Endalus Wrote: You are not allowed to view links. Register or Login to view.I remember someone posted some python code that would run a backup whenever the model saved in the DFL github repo. Unfortunately it was in one of the Free For Talk issue threads that seem to have been deleted now. Unfortunately I don't know enough python to write it from scratch.
In my personal copy I've actually changed 900 (15 minutes) to 910 to help account for collision. On average the backup will lag behind the save by an additional 40 seconds every hour, but you don't have to worry about it for almost a full day. I hit save and reset the backup script when I leave for work and when I get home from dinner and it keeps pretty well synched.

Just run the .bat file as part of the save model function.

I don't know anything about python either, there's probably a better way to do this, but I just:

import os

at the beginning of trainer.py, then at the end of logical if: "if not is_reached_goal and (time.time() - last_save_time) >= save_interval_min*60:" run

os.system ("backupmodel.bat")

Setup .bat file to copy model to wherever you want.  I copy to my nas as it has basically unlimited space available.  I've previously posted the backup .bat file I use.  Manual cleanup of backups is required, which I'm lazy about.  For example I've got 95 model backups of Liza Soberano, which I probably don't need, as nothing's gone wrong.  But it's only 60 gigs, so whatever.
#7
[Image: DQTTwB8h.png]


To enter manually ?

Can't it be fully automatic?
#8
(05-26-2019, 03:34 AM)a0933974848 Wrote: You are not allowed to view links. Register or Login to view.[Image: DQTTwB8h.png]


To enter manually ?

Can't it be fully automatic?

Why are you running that code manually? You're supposed to copy and paste it into a bat file.

Forum Jump:

Users browsing this thread: 1 Guest(s)