MrDeepFakes Forums
  • New and improved dark forum theme!
  • Guests can now comment on videos on the tube.
   
sakajoSOLVED Optimizer 1 and lower batch VS. Optimizer 2 and higher batch
#1
Which is a better, using Optimizer 1 with a lower batch size, or optimizer 2 with a higher batch size? 

Still trying to grasp the idea of batch sizes. Is larger always better?
#2
(01-15-2020, 01:26 AM)sakajo Wrote: You are not allowed to view links. Register or Login to view.Which is a better, using Optimizer 1 with a lower batch size, or optimizer 2 with a higher batch size? 
Still trying to grasp the idea of batch sizes. Is larger always better?

Hard to answer.  Its not really a comparison.  Depends on your batch size.
Obviously lower optimizer is faster since no off loading to RAM.  However, if you have to lower your batch size from say 10 to 4, probably not worth it in the later stages of training.  Would be fine in the beginning stages.   

tl;dr - lower optimizer is technically faster, but not necessarily better.  If you dont have a large enough batch size, you'll never get your loss counts down.
#3
(01-15-2020, 01:44 AM)avalentino93 Wrote: You are not allowed to view links. Register or Login to view.
(01-15-2020, 01:26 AM)sakajo Wrote: You are not allowed to view links. Register or Login to view.Which is a better, using Optimizer 1 with a lower batch size, or optimizer 2 with a higher batch size? 
Still trying to grasp the idea of batch sizes. Is larger always better?

Hard to answer.  Its not really a comparison.  Depends on your batch size.
Obviously lower optimizer is faster since no off loading to RAM.  However, if you have to lower your batch size from say 10 to 4, probably not worth it in the later stages of training.  Would be fine in the beginning stages.   

tl;dr - lower optimizer is technically faster, but not necessarily better.  If you dont have a large enough batch size, you'll never get your loss counts down.

Im looking at Opt2=BS 10, and Opt 1=BS 7. 

I get the feeling that the preview looks worse after switching to Opt 1, but not entirely sure. The blue/yellow graphs seems to be longer (or wider) since I switched. What does that mean?
#4
Assuming

Opt1, BS 7 = 2000 ms then 2000/7 = 285.71 ms for 1 batch size
and
Opt2, BS 10 = 2500 ms then 2500/10 = 250 ms for 1 batch size

therefore Opt2 will be faster since more data is being fed into the network
#5
(01-15-2020, 01:57 AM)Fappenberg Wrote: You are not allowed to view links. Register or Login to view.Assuming

Opt1, BS 7 = 2000 ms then 2000/7 = 285.71 ms for 1 batch size
and
Opt2, BS 10 = 2500 ms then 2500/10 = 250 ms for 1 batch size

therefore Opt2 will be faster since more data is being fed into the network

For me its roughly the same

Opt1, BS 7 = 1090 ms then 1090/7 = 155.7 ms for 1 batch size
and
Opt2, BS 10 =  ms then 1600/10 = 160 ms for 1 batch size

My issue is not with speed, its with quality. If im not mistaken the higher batch size means better final quality of the result?
#6
Generally yes, higher batch will give a more accurate result.
#7
Opt1 for the first x00,000 iterations
Op2 from there on out
#8
Learning with a big batch will give you a more accurate learning vector.

That is, if you set the maximum possible, the result will be better in the end. But also you will get a very long training time. Therefore, I agree with @Fappenberg , we need a middle ground between the size of the batch and the speed of training.

If speed is important to you - train with batch 4 or even lower.
If you need quality and you absolutely do not care how long it takes - set the maximum possible batch.
If you want to quickly and efficiently, use the formula provided by Fappenberg.
#9
(01-17-2020, 12:04 AM)VladShev Wrote: You are not allowed to view links. Register or Login to view.Learning with a big batch will give you a more accurate learning vector.

That is, if you set the maximum possible, the result will be better in the end. But also you will get a very long training time. Therefore, I agree with @Fappenberg , we need a middle ground between the size of the batch and the speed of training.

If speed is important to you - train with batch 4 or even lower.
If you need quality and you absolutely do not care how long it takes - set the maximum possible batch.
If you want to quickly and efficiently, use the formula provided by Fappenberg.

And if im opting for quality i should train highest batch size possible from the start?
#10
(01-17-2020, 12:38 AM)sakajo Wrote: You are not allowed to view links. Register or Login to view.
(01-17-2020, 12:04 AM)VladShev Wrote: You are not allowed to view links. Register or Login to view.Learning with a big batch will give you a more accurate learning vector.

That is, if you set the maximum possible, the result will be better in the end. But also you will get a very long training time. Therefore, I agree with @Fappenberg , we need a middle ground between the size of the batch and the speed of training.

If speed is important to you - train with batch 4 or even lower.
If you need quality and you absolutely do not care how long it takes - set the maximum possible batch.
If you want to quickly and efficiently, use the formula provided by Fappenberg.

And if im opting for quality i should train highest batch size possible from the start?

Opinions differ, someone believes that it’s better to set the batch low at the beginning, and then increase it, someone who needs to put the maximum possible batch immediately and so on all the time.

My personal opinion is that if the goal is to achieve maximum quality, then you need to set the maximum batch at the beginning. A more accurate direction of study, therefore, a more accurate model. I myself did not work with machine learning, I only read articles on this topic, so that I may be wrong in this matter. Ideally, of course, hear the opinion of the real data scientist.

Forum Jump:

Users browsing this thread: 1 Guest(s)