12345678910111213141516171819202122232425 |
- Learning Rate Schedulers
- ===================
- DeepSpeed offers implementations of ``LRRangeTest``, ``OneCycle``, ``WarmupLR``, ``WarmupDecayLR`` learning rate schedulers. When using a DeepSpeed's learning rate scheduler (specified in the `ds_config.json` file), DeepSpeed calls the `step()` method of the scheduler at every training step (when `model_engine.step()` is executed). When not using a DeepSpeed's learning rate scheduler:
- * if the schedule is supposed to execute at every training step, then the user can pass the scheduler to `deepspeed.initialize` when initializing the DeepSpeed engine and let DeepSpeed manage it for update or save/restore.
- * if the schedule is supposed to execute at any other interval (e.g., training epochs), then the user should NOT pass the scheduler to DeepSpeed during initialization and must manage it explicitly.
- LRRangeTest
- ---------------------------
- .. autoclass:: deepspeed.runtime.lr_schedules.LRRangeTest
- OneCycle
- ---------------------------
- .. autoclass:: deepspeed.runtime.lr_schedules.OneCycle
- WarmupLR
- ---------------------------
- .. autoclass:: deepspeed.runtime.lr_schedules.WarmupLR
- WarmupDecayLR
- ---------------------------
- .. autoclass:: deepspeed.runtime.lr_schedules.WarmupDecayLR
|