ignite.contrib.engines

Contribution module of engines and helper tools

Truncated Backpropagation Throught Time

class ignite.contrib.engines.tbptt.Tbptt_Events(value)

Aditional tbptt events.

Additional events for truncated backpropagation throught time dedicated trainer.

ignite.contrib.engines.tbptt.create_supervised_tbptt_trainer(model, optimizer, loss_fn, tbtt_step, dim=0, device=None, non_blocking=False, prepare_batch=<function _prepare_batch>)

Create a trainer for truncated backprop through time supervised models.

Training recurrent model on long sequences is computationally intensive as it requires to process the whole sequence before getting a gradient. However, when the training loss is computed over many outputs (X to many), there is an opportunity to compute a gradient over a subsequence. This is known as truncated backpropagation through time. This supervised trainer apply gradient optimization step every tbtt_step time steps of the sequence, while backpropagating through the same tbtt_step time steps.

Parameters

• model (torch.nn.Module) – the model to train.

• optimizer (torch.optim.Optimizer) – the optimizer to use.

• loss_fn (torch.nn loss function) – the loss function to use.

• tbtt_step (int) – the length of time chunks (last one may be smaller).

• dim (int) – axis representing the time dimension.

• device (str, optional) – device type specification (default: None). Applies to batches.

• non_blocking (bool, optional) – if True and this copy is between CPU and GPU, the copy may occur asynchronously with respect to the host. For other cases, this argument has no effect.

• prepare_batch (callable, optional) – function that receives batch, device, non_blocking and outputs tuple of tensors (batch_x, batch_y).

Warning

The internal use of device has changed. device will now only be used to move the input data to the correct device. The model should be moved by the user before creating an optimizer.

For more information see:

• PyTorch Documentation

• PyTorch’s Explanation

Returns

a trainer engine with supervised update function.

Return type

Engine

Helper methods to setup trainer/evaluator

ignite.contrib.engines.common.add_early_stopping_by_val_score(patience, evaluator, trainer, metric_name)

Method setups early stopping handler based on the score (named by metric_name) provided by evaluator.

Parameters

• patience (int) – number of events to wait if no improvement and then stop the training.

• evaluator (Engine) – evaluation engine used to provide the score

• trainer (Engine) – trainer engine to stop the run if no improvement.

• metric_name (str) – metric name to use for score evaluation. This metric should be present in evaluator.state.metrics.

Returns

A EarlyStopping handler.

ignite.contrib.engines.common.save_best_model_by_val_score(output_path, evaluator, model, metric_name, n_saved=3, trainer=None, tag='val')

Method adds a handler to evaluator to save best models based on the score (named by metric_name) provided by evaluator.

Parameters

• output_path (str) – output path to indicate where to save best models

• evaluator (Engine) – evaluation engine used to provide the score

• model (nn.Module) – model to store

• metric_name (str) – metric name to use for score evaluation. This metric should be present in evaluator.state.metrics.

• n_saved (int, optional) – number of best models to store

• trainer (Engine, optional) – trainer engine to fetch the epoch when saving the best model.

• tag (str, optional) – score name prefix: {tag}_{metric_name}. By default, tag is “val”.

Returns

A Checkpoint handler.

ignite.contrib.engines.common.setup_common_distrib_training_handlers(trainer, train_sampler=None, to_save=None, save_every_iters=1000, output_path=None, lr_scheduler=None, with_gpu_stats=False, output_names=None, with_pbars=True, with_pbar_on_iters=True, log_every_iters=100, device=None, stop_on_nan=True, clear_cuda_cache=True)

Helper method to setup trainer with common handlers (it also supports distributed configuration):

• TerminateOnNan

• handler to setup learning rate scheduling

• ModelCheckpoint

• RunningAverage on update_function output

• Two progress bars on epochs and optionally on iterations

Parameters

• trainer (Engine) – trainer engine. Output of trainer’s update_function should be a dictionary or sequence or a single tensor.

• train_sampler (torch.utils.data.DistributedSampler, optional) – Optional distributed sampler used to call set_epoch method on epoch started event.

• to_save (dict, optional) – dictionary with objects to save in the checkpoint. This argument is passed to Checkpoint instance.

• save_every_iters (int, optional) – saving interval. By default, to_save objects are stored each 1000 iterations.

• output_path (str, optional) – output path to indicate where to_save objects are stored.

• lr_scheduler (ParamScheduler or subclass of torch.optim.lr_scheduler._LRScheduler) – learning rate scheduler as native torch LRScheduler or ignite’s parameter scheduler.

• with_gpu_stats (bool, optional) – if True, GpuInfo is attached to the trainer. This requires pynvml package to be installed.

• output_names (list/tuple, optional) – list of names associated with update_function output dictionary.

• with_pbars (bool, optional) – if True, two progress bars on epochs and optionally on iterations are attached. Default, True.

• with_pbar_on_iters (bool, optional) – if True, a progress bar on iterations is attached to the trainer. Default, True.

• log_every_iters (int, optional) – logging interval for GpuInfo and for epoch-wise progress bar. Default, 100.

• stop_on_nan (bool, optional) – if True, TerminateOnNan handler is added to the trainer. Default, True.

• clear_cuda_cache (bool, optional) – if True, torch.cuda.empty_cache() is called every end of epoch. Default, True.

• device (str of torch.device, optional) – deprecated argument, it will be removed in v0.5.0.

ignite.contrib.engines.common.setup_common_training_handlers(trainer, train_sampler=None, to_save=None, save_every_iters=1000, output_path=None, lr_scheduler=None, with_gpu_stats=False, output_names=None, with_pbars=True, with_pbar_on_iters=True, log_every_iters=100, device=None, stop_on_nan=True, clear_cuda_cache=True)

Helper method to setup trainer with common handlers (it also supports distributed configuration):

• TerminateOnNan

• handler to setup learning rate scheduling

• ModelCheckpoint

• RunningAverage on update_function output

• Two progress bars on epochs and optionally on iterations

Parameters

• trainer (Engine) – trainer engine. Output of trainer’s update_function should be a dictionary or sequence or a single tensor.

• train_sampler (torch.utils.data.DistributedSampler, optional) – Optional distributed sampler used to call set_epoch method on epoch started event.

• to_save (dict, optional) – dictionary with objects to save in the checkpoint. This argument is passed to Checkpoint instance.

• save_every_iters (int, optional) – saving interval. By default, to_save objects are stored each 1000 iterations.

• output_path (str, optional) – output path to indicate where to_save objects are stored.

• lr_scheduler (ParamScheduler or subclass of torch.optim.lr_scheduler._LRScheduler) – learning rate scheduler as native torch LRScheduler or ignite’s parameter scheduler.

• with_gpu_stats (bool, optional) – if True, GpuInfo is attached to the trainer. This requires pynvml package to be installed.

• output_names (list/tuple, optional) – list of names associated with update_function output dictionary.

• with_pbars (bool, optional) – if True, two progress bars on epochs and optionally on iterations are attached. Default, True.

• with_pbar_on_iters (bool, optional) – if True, a progress bar on iterations is attached to the trainer. Default, True.

• log_every_iters (int, optional) – logging interval for GpuInfo and for epoch-wise progress bar. Default, 100.

• stop_on_nan (bool, optional) – if True, TerminateOnNan handler is added to the trainer. Default, True.

• clear_cuda_cache (bool, optional) – if True, torch.cuda.empty_cache() is called every end of epoch. Default, True.

• device (str of torch.device, optional) – deprecated argument, it will be removed in v0.5.0.

ignite.contrib.engines.common.setup_mlflow_logging(trainer, optimizers=None, evaluators=None, log_every_iters=100, **kwargs)

Method to setup MLflow logging on trainer and a list of evaluators. Logged metrics are:

• Training metrics, e.g. running average loss values

• Learning rate(s)

• Evaluation metrics

Parameters

• trainer (Engine) – trainer engine

• optimizers (torch.optim.Optimizer or dict of torch.optim.Optimizer, optional) – single or dictionary of torch optimizers. If a dictionary, keys are used as tags arguments for logging.

• evaluators (Engine or dict of Engine, optional) – single or dictionary of evaluators. If a dictionary, keys are used as tags arguments for logging.

• log_every_iters (int, optional) – interval for loggers attached to iteration events. To log every iteration, value can be set to 1 or None.

• **kwargs – optional keyword args to be passed to construct the logger.

Returns

MLflowLogger

ignite.contrib.engines.common.setup_neptune_logging(trainer, optimizers=None, evaluators=None, log_every_iters=100, **kwargs)

Method to setup Neptune logging on trainer and a list of evaluators. Logged metrics are:

• Training metrics, e.g. running average loss values

• Learning rate(s)

• Evaluation metrics

Parameters

• trainer (Engine) – trainer engine

• optimizers (torch.optim.Optimizer or dict of torch.optim.Optimizer, optional) – single or dictionary of torch optimizers. If a dictionary, keys are used as tags arguments for logging.

• evaluators (Engine or dict of Engine, optional) – single or dictionary of evaluators. If a dictionary, keys are used as tags arguments for logging.

• log_every_iters (int, optional) – interval for loggers attached to iteration events. To log every iteration, value can be set to 1 or None.

• **kwargs – optional keyword args to be passed to construct the logger.

Returns

NeptuneLogger

ignite.contrib.engines.common.setup_plx_logging(trainer, optimizers=None, evaluators=None, log_every_iters=100, **kwargs)

Method to setup Polyaxon logging on trainer and a list of evaluators. Logged metrics are:

• Training metrics, e.g. running average loss values

• Learning rate(s)

• Evaluation metrics

Parameters

• trainer (Engine) – trainer engine

• optimizers (torch.optim.Optimizer or dict of torch.optim.Optimizer, optional) – single or dictionary of torch optimizers. If a dictionary, keys are used as tags arguments for logging.

• evaluators (Engine or dict of Engine, optional) – single or dictionary of evaluators. If a dictionary, keys are used as tags arguments for logging.

• log_every_iters (int, optional) – interval for loggers attached to iteration events. To log every iteration, value can be set to 1 or None.

• **kwargs – optional keyword args to be passed to construct the logger.

Returns

PolyaxonLogger

ignite.contrib.engines.common.setup_tb_logging(output_path, trainer, optimizers=None, evaluators=None, log_every_iters=100, **kwargs)

Method to setup TensorBoard logging on trainer and a list of evaluators. Logged metrics are:

• Training metrics, e.g. running average loss values

• Learning rate(s)

• Evaluation metrics

Parameters

• output_path (str) – logging directory path

• trainer (Engine) – trainer engine

• optimizers (torch.optim.Optimizer or dict of torch.optim.Optimizer, optional) – single or dictionary of torch optimizers. If a dictionary, keys are used as tags arguments for logging.

• evaluators (Engine or dict of Engine, optional) – single or dictionary of evaluators. If a dictionary, keys are used as tags arguments for logging.

• log_every_iters (int, optional) – interval for loggers attached to iteration events. To log every iteration, value can be set to 1 or None.

• **kwargs – optional keyword args to be passed to construct the logger.

Returns

TensorboardLogger

ignite.contrib.engines.common.setup_trains_logging(trainer, optimizers=None, evaluators=None, log_every_iters=100, **kwargs)

Method to setup Trains logging on trainer and a list of evaluators. Logged metrics are:

• Training metrics, e.g. running average loss values

• Learning rate(s)

• Evaluation metrics

Parameters

• trainer (Engine) – trainer engine

• optimizers (torch.optim.Optimizer or dict of torch.optim.Optimizer, optional) – single or dictionary of torch optimizers. If a dictionary, keys are used as tags arguments for logging.

• evaluators (Engine or dict of Engine, optional) – single or dictionary of evaluators. If a dictionary, keys are used as tags arguments for logging.

• log_every_iters (int, optional) – interval for loggers attached to iteration events. To log every iteration, value can be set to 1 or None.

• **kwargs – optional keyword args to be passed to construct the logger.

Returns

TrainsLogger

ignite.contrib.engines.common.setup_visdom_logging(trainer, optimizers=None, evaluators=None, log_every_iters=100, **kwargs)

Method to setup Visdom logging on trainer and a list of evaluators. Logged metrics are:

• Training metrics, e.g. running average loss values

• Learning rate(s)

• Evaluation metrics

Parameters

• trainer (Engine) – trainer engine

• optimizers (torch.optim.Optimizer or dict of torch.optim.Optimizer, optional) – single or dictionary of torch optimizers. If a dictionary, keys are used as tags arguments for logging.

• evaluators (Engine or dict of Engine, optional) – single or dictionary of evaluators. If a dictionary, keys are used as tags arguments for logging.

• log_every_iters (int, optional) – interval for loggers attached to iteration events. To log every iteration, value can be set to 1 or None.

• **kwargs – optional keyword args to be passed to construct the logger.

Returns

VisdomLogger

ignite.contrib.engines.common.setup_wandb_logging(trainer, optimizers=None, evaluators=None, log_every_iters=100, **kwargs)

Method to setup WandB logging on trainer and a list of evaluators. Logged metrics are:

• Training metrics, e.g. running average loss values

• Learning rate(s)

• Evaluation metrics

Parameters

• trainer (Engine) – trainer engine

• optimizers (torch.optim.Optimizer or dict of torch.optim.Optimizer, optional) – single or dictionary of torch optimizers. If a dictionary, keys are used as tags arguments for logging.

• evaluators (Engine or dict of Engine, optional) – single or dictionary of evaluators. If a dictionary, keys are used as tags arguments for logging.

• log_every_iters (int, optional) – interval for loggers attached to iteration events. To log every iteration, value can be set to 1 or None.

• **kwargs – optional keyword args to be passed to construct the logger.

Returns

WandBLogger