Shortcuts

mlflow_logger#

MLflow logger and its helper handlers.

Classes

MLflowLogger

MLflow tracking client handler to log parameters and metrics during the training and validation.

OptimizerParamsHandler

Helper handler to log optimizer parameters

OutputHandler

Helper handler to log engine's output and/or metrics.

class ignite.handlers.mlflow_logger.MLflowLogger(tracking_uri=None)[source]#

MLflow tracking client handler to log parameters and metrics during the training and validation.

This class requires mlflow package to be installed:

pip install mlflow
Parameters

tracking_uri (Optional[str]) – MLflow tracking uri. See MLflow docs for more details

Examples

from ignite.handlers.mlflow_logger import *

# Create a logger
mlflow_logger = MLflowLogger()

# Log experiment parameters:
mlflow_logger.log_params({
    "seed": seed,
    "batch_size": batch_size,
    "model": model.__class__.__name__,

    "pytorch version": torch.__version__,
    "ignite version": ignite.__version__,
    "cuda version": torch.version.cuda,
    "device name": torch.cuda.get_device_name(0)
})

# Attach the logger to the trainer to log training loss at each iteration
mlflow_logger.attach_output_handler(
    trainer,
    event_name=Events.ITERATION_COMPLETED,
    tag="training",
    output_transform=lambda loss: {'loss': loss}
)

# Attach the logger to the evaluator on the training dataset and log NLL, Accuracy metrics after each epoch
# We setup `global_step_transform=global_step_from_engine(trainer)` to take the epoch
# of the `trainer` instead of `train_evaluator`.
mlflow_logger.attach_output_handler(
    train_evaluator,
    event_name=Events.EPOCH_COMPLETED,
    tag="training",
    metric_names=["nll", "accuracy"],
    global_step_transform=global_step_from_engine(trainer),
)

# Attach the logger to the evaluator on the validation dataset and log NLL, Accuracy metrics after
# each epoch. We setup `global_step_transform=global_step_from_engine(trainer)` to take the epoch of the
# `trainer` instead of `evaluator`.
mlflow_logger.attach_output_handler(
    evaluator,
    event_name=Events.EPOCH_COMPLETED,
    tag="validation",
    metric_names=["nll", "accuracy"],
    global_step_transform=global_step_from_engine(trainer)),
)

# Attach the logger to the trainer to log optimizer's parameters, e.g. learning rate at each iteration
mlflow_logger.attach_opt_params_handler(
    trainer,
    event_name=Events.ITERATION_STARTED,
    optimizer=optimizer,
    param_name='lr'  # optional
)
class ignite.handlers.mlflow_logger.OptimizerParamsHandler(optimizer, param_name='lr', tag=None)[source]#

Helper handler to log optimizer parameters

Parameters
  • optimizer (Optimizer) – torch optimizer or any object with attribute param_groups as a sequence.

  • param_name (str) – parameter name

  • tag (Optional[str]) – common title for all produced plots. For example, ‘generator’

Examples

from ignite.handlers.mlflow_logger import *

# Create a logger
mlflow_logger = MLflowLogger()
# Optionally, user can specify tracking_uri with corresponds to MLFLOW_TRACKING_URI
# mlflow_logger = MLflowLogger(tracking_uri="uri")

# Attach the logger to the trainer to log optimizer's parameters, e.g. learning rate at each iteration
mlflow_logger.attach(
    trainer,
    log_handler=OptimizerParamsHandler(optimizer),
    event_name=Events.ITERATION_STARTED
)
# or equivalently
mlflow_logger.attach_opt_params_handler(
    trainer,
    event_name=Events.ITERATION_STARTED,
    optimizer=optimizer
)
class ignite.handlers.mlflow_logger.OutputHandler(tag, metric_names=None, output_transform=None, global_step_transform=None, state_attributes=None)[source]#

Helper handler to log engine’s output and/or metrics.

Parameters
  • tag (str) – common title for all produced plots. For example, ‘training’

  • metric_names (Optional[Union[str, List[str]]]) – list of metric names to plot or a string “all” to plot all available metrics.

  • output_transform (Optional[Callable]) – output transform function to prepare engine.state.output as a number. For example, output_transform = lambda output: output This function can also return a dictionary, e.g {‘loss’: loss1, ‘another_loss’: loss2} to label the plot with corresponding keys.

  • global_step_transform (Optional[Callable[[Engine, Union[str, Events]], int]]) – global step transform function to output a desired global step. Input of the function is (engine, event_name). Output of function should be an integer. Default is None, global_step based on attached engine. If provided, uses function output as global_step. To setup global step from another engine, please use global_step_from_engine().

  • state_attributes (Optional[List[str]]) – list of attributes of the trainer.state to plot.

Examples

from ignite.handlers.mlflow_logger import *

# Create a logger
mlflow_logger = MLflowLogger()

# Attach the logger to the evaluator on the validation dataset and log NLL, Accuracy metrics after
# each epoch. We setup `global_step_transform=global_step_from_engine(trainer)` to take the epoch
# of the `trainer`:
mlflow_logger.attach(
    evaluator,
    log_handler=OutputHandler(
        tag="validation",
        metric_names=["nll", "accuracy"],
        global_step_transform=global_step_from_engine(trainer)
    ),
    event_name=Events.EPOCH_COMPLETED
)
# or equivalently
mlflow_logger.attach_output_handler(
    evaluator,
    event_name=Events.EPOCH_COMPLETED,
    tag="validation",
    metric_names=["nll", "accuracy"],
    global_step_transform=global_step_from_engine(trainer)
)

Another example, where model is evaluated every 500 iterations:

from ignite.handlers.mlflow_logger import *

@trainer.on(Events.ITERATION_COMPLETED(every=500))
def evaluate(engine):
    evaluator.run(validation_set, max_epochs=1)

mlflow_logger = MLflowLogger()

def global_step_transform(*args, **kwargs):
    return trainer.state.iteration

# Attach the logger to the evaluator on the validation dataset and log NLL, Accuracy metrics after
# every 500 iterations. Since evaluator engine does not have access to the training iteration, we
# provide a global_step_transform to return the trainer.state.iteration for the global_step, each time
# evaluator metrics are plotted on MLflow.

mlflow_logger.attach_output_handler(
    evaluator,
    event_name=Events.EPOCH_COMPLETED,
    tag="validation",
    metrics=["nll", "accuracy"],
    global_step_transform=global_step_transform
)

Another example where the State Attributes trainer.state.alpha and trainer.state.beta are also logged along with the NLL and Accuracy after each iteration:

mlflow_logger.attach_output_handler(
    trainer,
    event_name=Events.ITERATION_COMPLETED,
    tag="training",
    metrics=["nll", "accuracy"],
    state_attributes=["alpha", "beta"],
)

Example of global_step_transform:

def global_step_transform(engine, event_name):
    return engine.state.get_event_attrib_value(event_name)

Changed in version 0.4.7: accepts an optional list of state_attributes

ignite.handlers.mlflow_logger.global_step_from_engine(engine, custom_event_name=None)[source]#

Helper method to setup global_step_transform function using another engine. This can be helpful for logging trainer epoch/iteration while output handler is attached to an evaluator.

Parameters
  • engine (Engine) – engine which state is used to provide the global step

  • custom_event_name (Optional[Events]) – registered event name. Optional argument, event name to use.

Returns

global step based on provided engine

Return type

Callable