mlflow_logger#

MLflow logger and its helper handlers.

Classes

`MLflowLogger`	MLflow tracking client handler to log parameters and metrics during the training and validation.
`OptimizerParamsHandler`	Helper handler to log optimizer parameters
`OutputHandler`	Helper handler to log engine's output and/or metrics.

class ignite.contrib.handlers.mlflow_logger.MLflowLogger(tracking_uri=None)[source]#

MLflow tracking client handler to log parameters and metrics during the training and validation.

This class requires mlflow package to be installed:

pip install mlflow

Parameters: tracking_uri (Optional[str]) – MLflow tracking uri. See MLflow docs for more details

Examples

from ignite.contrib.handlers.mlflow_logger import *

# Create a logger
mlflow_logger = MLflowLogger()

# Log experiment parameters:
mlflow_logger.log_params({
    "seed": seed,
    "batch_size": batch_size,
    "model": model.__class__.__name__,

    "pytorch version": torch.__version__,
    "ignite version": ignite.__version__,
    "cuda version": torch.version.cuda,
    "device name": torch.cuda.get_device_name(0)
})

# Attach the logger to the trainer to log training loss at each iteration
mlflow_logger.attach_output_handler(
    trainer,
    event_name=Events.ITERATION_COMPLETED,
    tag="training",
    output_transform=lambda loss: {'loss': loss}
)

# Attach the logger to the evaluator on the training dataset and log NLL, Accuracy metrics after each epoch
# We setup `global_step_transform=global_step_from_engine(trainer)` to take the epoch
# of the `trainer` instead of `train_evaluator`.
mlflow_logger.attach_output_handler(
    train_evaluator,
    event_name=Events.EPOCH_COMPLETED,
    tag="training",
    metric_names=["nll", "accuracy"],
    global_step_transform=global_step_from_engine(trainer),
)

# Attach the logger to the evaluator on the validation dataset and log NLL, Accuracy metrics after
# each epoch. We setup `global_step_transform=global_step_from_engine(trainer)` to take the epoch of the
# `trainer` instead of `evaluator`.
mlflow_logger.attach_output_handler(
    evaluator,
    event_name=Events.EPOCH_COMPLETED,
    tag="validation",
    metric_names=["nll", "accuracy"],
    global_step_transform=global_step_from_engine(trainer)),
)

# Attach the logger to the trainer to log optimizer's parameters, e.g. learning rate at each iteration
mlflow_logger.attach_opt_params_handler(
    trainer,
    event_name=Events.ITERATION_STARTED,
    optimizer=optimizer,
    param_name='lr'  # optional
)

class ignite.contrib.handlers.mlflow_logger.OptimizerParamsHandler(optimizer, param_name='lr', tag=None)[source]#

Helper handler to log optimizer parameters

Parameters

optimizer (Optimizer) – torch optimizer or any object with attribute param_groups as a sequence.
param_name (str) – parameter name
tag (Optional[str]) – common title for all produced plots. For example, ‘generator’

Examples

from ignite.contrib.handlers.mlflow_logger import *

# Create a logger
mlflow_logger = MLflowLogger()
# Optionally, user can specify tracking_uri with corresponds to MLFLOW_TRACKING_URI
# mlflow_logger = MLflowLogger(tracking_uri="uri")

# Attach the logger to the trainer to log optimizer's parameters, e.g. learning rate at each iteration
mlflow_logger.attach(
    trainer,
    log_handler=OptimizerParamsHandler(optimizer),
    event_name=Events.ITERATION_STARTED
)
# or equivalently
mlflow_logger.attach_opt_params_handler(
    trainer,
    event_name=Events.ITERATION_STARTED,
    optimizer=optimizer
)

class ignite.contrib.handlers.mlflow_logger.OutputHandler(tag, metric_names=None, output_transform=None, global_step_transform=None, state_attributes=None)[source]#

Helper handler to log engine’s output and/or metrics.

Parameters

tag (str) – common title for all produced plots. For example, ‘training’
metric_names (Optional[Union[str, List[str]]]) – list of metric names to plot or a string “all” to plot all available metrics.
output_transform (Optional[Callable]) – output transform function to prepare engine.state.output as a number. For example, output_transform = lambda output: output This function can also return a dictionary, e.g {‘loss’: loss1, ‘another_loss’: loss2} to label the plot with corresponding keys.
global_step_transform (Optional[Callable]) – global step transform function to output a desired global step. Input of the function is (engine, event_name). Output of function should be an integer. Default is None, global_step based on attached engine. If provided, uses function output as global_step. To setup global step from another engine, please use global_step_from_engine().
state_attributes (Optional[List[str]]) – list of attributes of the trainer.state to plot.

Examples

from ignite.contrib.handlers.mlflow_logger import *

# Create a logger
mlflow_logger = MLflowLogger()

# Attach the logger to the evaluator on the validation dataset and log NLL, Accuracy metrics after
# each epoch. We setup `global_step_transform=global_step_from_engine(trainer)` to take the epoch
# of the `trainer`:
mlflow_logger.attach(
    evaluator,
    log_handler=OutputHandler(
        tag="validation",
        metric_names=["nll", "accuracy"],
        global_step_transform=global_step_from_engine(trainer)
    ),
    event_name=Events.EPOCH_COMPLETED
)
# or equivalently
mlflow_logger.attach_output_handler(
    evaluator,
    event_name=Events.EPOCH_COMPLETED,
    tag="validation",
    metric_names=["nll", "accuracy"],
    global_step_transform=global_step_from_engine(trainer)
)

Another example, where model is evaluated every 500 iterations:

from ignite.contrib.handlers.mlflow_logger import *

@trainer.on(Events.ITERATION_COMPLETED(every=500))
def evaluate(engine):
    evaluator.run(validation_set, max_epochs=1)

mlflow_logger = MLflowLogger()

def global_step_transform(*args, **kwargs):
    return trainer.state.iteration

# Attach the logger to the evaluator on the validation dataset and log NLL, Accuracy metrics after
# every 500 iterations. Since evaluator engine does not have access to the training iteration, we
# provide a global_step_transform to return the trainer.state.iteration for the global_step, each time
# evaluator metrics are plotted on MLflow.

mlflow_logger.attach_output_handler(
    evaluator,
    event_name=Events.EPOCH_COMPLETED,
    tag="validation",
    metrics=["nll", "accuracy"],
    global_step_transform=global_step_transform
)

Another example where the State Attributes trainer.state.alpha and trainer.state.beta are also logged along with the NLL and Accuracy after each iteration:

mlflow_logger.attach_output_handler(
    trainer,
    event_name=Events.ITERATION_COMPLETED,
    tag="training",
    metrics=["nll", "accuracy"],
    state_attributes=["alpha", "beta"],
)

Example of global_step_transform:

def global_step_transform(engine, event_name):
    return engine.state.get_event_attrib_value(event_name)

Changed in version 0.4.7: accepts an optional list of state_attributes

ignite.contrib.handlers.mlflow_logger.global_step_from_engine(engine, custom_event_name=None)[source]#

Helper method to setup global_step_transform function using another engine. This can be helpful for logging trainer epoch/iteration while output handler is attached to an evaluator.

Parameters

engine (Engine) – engine which state is used to provide the global step
custom_event_name (Optional[Events]) – registered event name. Optional argument, event name to use.

Returns

global step based on provided engine

Return type

Callable