199. Simple SGD vs Cyclic Learning Rate

Simple SGD vs Cyclic Learning Rate

I compared the training speed between two optimizers by training a UNet Model.

  • Simple SGD 
    optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
  • Cyclic Learning Rate 
    optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
scheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=0.1, max_lr=1e-4)

Cyclic Learning Rate was able to achieve lower loss and got to that level faster.

Related Posts