Shortcuts

FastaiLRFinder#

class ignite.handlers.lr_finder.FastaiLRFinder[source]#

Learning rate finder handler for supervised trainers.

While attached, the handler increases the learning rate in between two boundaries in a linear or exponential manner. It provides valuable information on how well the network can be trained over a range of learning rates and what can be an optimal learning rate.

Examples:

from ignite.handlers import FastaiLRFinder

trainer = ...
model = ...
optimizer = ...

lr_finder = FastaiLRFinder()
to_save = {"model": model, "optimizer": optimizer}

with lr_finder.attach(trainer, to_save=to_save) as trainer_with_lr_finder:
    trainer_with_lr_finder.run(dataloader)

# Get lr_finder results
lr_finder.get_results()

# Plot lr_finder results (requires matplotlib)
lr_finder.plot()

# get lr_finder suggestion for lr
lr_finder.lr_suggestion()

Note

When context manager is exited all LR finder’s handlers are removed.

Note

Please, also keep in mind that all other handlers attached the trainer will be executed during LR finder’s run.

Note

This class may require matplotlib package to be installed to plot learning rate range test:

pip install matplotlib

References

Cyclical Learning Rates for Training Neural Networks: https://arxiv.org/abs/1506.01186

fastai/lr_find: https://github.com/fastai/fastai

Methods

apply_suggested_lr

Applying the suggested learning rate(s) on the given optimizer.

attach

Attaches lr_finder to a given trainer.

get_results

returns

Dictionary with loss and lr logs from the previous run

lr_suggestion

returns

Learning rate at the minimum numerical gradient

plot

Plots the learning rate range test.

apply_suggested_lr(optimizer)[source]#

Applying the suggested learning rate(s) on the given optimizer.

Note

The given optimizer must be the same as the one we before found the suggested learning rate for.

Parameters

optimizer (Optimizer) – the optimizer to apply the suggested learning rate(s) on.

Return type

None

attach(trainer, to_save, output_transform=<function FastaiLRFinder.<lambda>>, num_iter=None, start_lr=None, end_lr=10.0, step_mode='exp', smooth_f=0.05, diverge_th=5.0)[source]#

Attaches lr_finder to a given trainer. It also resets model and optimizer at the end of the run.

Usage:

to_save = {"model": model, "optimizer": optimizer}
with lr_finder.attach(trainer, to_save=to_save) as trainer_with_lr_finder:
    trainer_with_lr_finder.run(dataloader)
Parameters
  • trainer (Engine) – lr_finder is attached to this trainer. Please, keep in mind that all attached handlers will be executed.

  • to_save (Mapping) – dictionary with optimizer and other objects that needs to be restored after running the LR finder. For example, to_save={'optimizer': optimizer, 'model': model}. It should contain “optimizer” key for the optimizer. Also all objects should implement state_dict and load_state_dict methods.

  • output_transform (Callable) – function that transforms the trainer’s state.output after each iteration. It must return the loss of that iteration.

  • num_iter (Optional[int]) – number of iterations for lr schedule between base lr and end_lr. Default, it will run for trainer.state.epoch_length * trainer.state.max_epochs.

  • start_lr (Optional[float]) – lower bound for lr search. Default, Learning Rate specified with the optimizer.

  • end_lr (float) – upper bound for lr search. Default, 10.0.

  • step_mode (str) – “exp” or “linear”, which way should the lr be increased from start_lr to end_lr. Default, “exp”.

  • smooth_f (float) – loss smoothing factor in range [0, 1). Default, 0.05

  • diverge_th (float) – Used for stopping the search when current loss > diverge_th * best_loss. Default, 5.0.

Returns

trainer_with_lr_finder (trainer used for finding the lr)

Return type

Any

Note

lr_finder cannot be attached to more than one trainer at a time.

get_results()[source]#
Returns

Dictionary with loss and lr logs from the previous run

Return type

Dict[str, List[Any]]

lr_suggestion()[source]#
Returns

Learning rate at the minimum numerical gradient (ignoring the increasing part of the curve)

Return type

Any

plot(skip_start=10, skip_end=5, log_lr=True, display_suggestion=True, ax=None, **kwargs)[source]#

Plots the learning rate range test.

This method requires matplotlib package to be installed:

pip install matplotlib
Parameters
  • skip_start (int) – number of batches to trim from the start. Default: 10.

  • skip_end (int) – number of batches to trim from the start. Default: 5.

  • log_lr (bool) – True to plot the learning rate in a logarithmic scale; otherwise, plotted in a linear scale. Default: True.

  • display_suggestion (bool) – if True, red dot shows the suggested learning rate.

  • ax (Optional[Any]) – Pre-existing axes for the plot. Default: None.

  • kwargs (Any) – optional kwargs passed to plt.subplots if ax is not provided.

Return type

None

ax = lr_finder.plot(skip_end=0)
ax.figure.savefig("output.jpg")