neptune_logger#

Neptune logger and its helper handlers.

Classes

`GradsScalarHandler`	Helper handler to log model's gradients as scalars.
`NeptuneLogger`	Neptune handler to log metrics, model/optimizer parameters, gradients during the training and validation.
`NeptuneSaver`	Handler that saves input checkpoint to the Neptune server.
`OptimizerParamsHandler`	Helper handler to log optimizer parameters
`OutputHandler`	Helper handler to log engine's output and/or metrics
`WeightsScalarHandler`	Helper handler to log model's weights as scalars.

class ignite.contrib.handlers.neptune_logger.GradsScalarHandler(model, reduction=<function norm>, tag=None, whitelist=None)[source]#

Helper handler to log model’s gradients as scalars. Handler, upon construction, iterates over named parameters of the model and keep reference to ones permitted by the whitelist. Then at every call, applies reduction function to each parameter’s gradient, produces a scalar and logs it.

Parameters

model (Module) – model to log weights
reduction (Callable[[Tensor], Union[float, Tensor]]) – function to reduce parameters into scalar
tag (Optional[str]) – common title for all produced plots. For example, “generator”
whitelist (Optional[Union[List[str], Callable[[str, Parameter], bool]]]) – specific gradients to log. Should be list of model’s submodules or parameters names, or a callable which gets weight along with its name and determines if its gradient should be logged. Names should be fully-qualified. For more information please refer to PyTorch docs. If not given, all of model’s gradients are logged.

Examples

from ignite.contrib.handlers.neptune_logger import *

# Create a logger
# We are using the api_token for the anonymous user neptuner but you can use your own.

npt_logger = NeptuneLogger(
    api_token="ANONYMOUS",
    project_name="shared/pytorch-ignite-integration",
    experiment_name="cnn-mnist", # Optional,
    params={"max_epochs": 10}, # Optional,
    tags=["pytorch-ignite","minst"] # Optional
)

# Attach the logger to the trainer to log model's weights norm after each iteration
npt_logger.attach(
    trainer,
    event_name=Events.ITERATION_COMPLETED,
    log_handler=GradsScalarHandler(model, reduction=torch.norm)
)

from ignite.contrib.handlers.neptune_logger import *

npt_logger = NeptuneLogger(
    api_token="ANONYMOUS",
    project_name="shared/pytorch-ignite-integration",
    experiment_name="cnn-mnist", # Optional,
    params={"max_epochs": 10}, # Optional,
    tags=["pytorch-ignite","minst"] # Optional
)

# Log gradient of `base`
npt_logger.attach(
    trainer,
    event_name=Events.ITERATION_COMPLETED,
    log_handler=GradsScalarHandler(
        model,
        reduction=torch.norm,
        whitelist=['base']
    )
)

from ignite.contrib.handlers.neptune_logger import *

npt_logger = NeptuneLogger(
    api_token="ANONYMOUS",
    project_name="shared/pytorch-ignite-integration",
    experiment_name="cnn-mnist", # Optional,
    params={"max_epochs": 10}, # Optional,
    tags=["pytorch-ignite","minst"] # Optional
)

# Log gradient of weights which belong to a `fc` layer
def is_in_fc_layer(n, p):
    return 'fc' in n

npt_logger.attach(
    trainer,
    event_name=Events.ITERATION_COMPLETED,
    log_handler=GradsScalarHandler(model, whitelist=is_in_fc_layer)
)

Changed in version 0.4.9: optional argument whitelist added.

class ignite.contrib.handlers.neptune_logger.NeptuneLogger(*args, **kwargs)[source]#

Neptune handler to log metrics, model/optimizer parameters, gradients during the training and validation. It can also log model checkpoints to Neptune server.

pip install neptune-client

Parameters

api_token – Required in online mode. Neptune API token, found on https://neptune.ai.
project_name – Required in online mode. Qualified name of a project in a form of “namespace/project_name” for example “tom/minst-classification”. If None, the value of NEPTUNE_PROJECT environment variable will be taken. You need to create the project in https://neptune.ai first.
offline_mode – Optional default False. If offline_mode=True no logs will be send to neptune. Usually used for debug purposes.
experiment_name – Optional. Editable name of the experiment. Name is displayed in the experiment’s Details (Metadata section) and in experiments view as a column.
upload_source_files – Optional. List of source files to be uploaded. Must be list of str or single str. Uploaded sources are displayed in the experiment’s Source code tab. If None is passed, Python file from which experiment was created will be uploaded. Pass empty list ([]) to upload no files. Unix style pathname pattern expansion is supported. For example, you can pass *.py to upload all python source files from the current directory. For recursion lookup use **/*.py (for Python 3.5 and later). For more information see glob library.
params – Optional. Parameters of the experiment. After experiment creation params are read-only. Parameters are displayed in the experiment’s Parameters section and each key-value pair can be viewed in experiments view as a column.
properties – Optional default is {}. Properties of the experiment. They are editable after experiment is created. Properties are displayed in the experiment’s Details and each key-value pair can be viewed in experiments view as a column.
tags – Optional default []. Must be list of str. Tags of the experiment. Tags are displayed in the experiment’s Details and can be viewed in experiments view as a column.
args (Any) –
kwargs (Any) –

Examples

from ignite.contrib.handlers.neptune_logger import *

# Create a logger
# We are using the api_token for the anonymous user neptuner but you can use your own.

npt_logger = NeptuneLogger(
    api_token="ANONYMOUS",
    project_name="shared/pytorch-ignite-integration",
    experiment_name="cnn-mnist", # Optional,
    params={"max_epochs": 10}, # Optional,
    tags=["pytorch-ignite","minst"] # Optional
)

# Attach the logger to the trainer to log training loss at each iteration
npt_logger.attach_output_handler(
    trainer,
    event_name=Events.ITERATION_COMPLETED,
    tag="training",
    output_transform=lambda loss: {'loss': loss}
)

# Attach the logger to the evaluator on the training dataset and log NLL, Accuracy metrics after each epoch
# We setup `global_step_transform=global_step_from_engine(trainer)` to take the epoch
# of the `trainer` instead of `train_evaluator`.
npt_logger.attach_output_handler(
    train_evaluator,
    event_name=Events.EPOCH_COMPLETED,
    tag="training",
    metric_names=["nll", "accuracy"],
    global_step_transform=global_step_from_engine(trainer),
)

# Attach the logger to the evaluator on the validation dataset and log NLL, Accuracy metrics after
# each epoch. We setup `global_step_transform=global_step_from_engine(trainer)` to take the epoch of the
# `trainer` instead of `evaluator`.
npt_logger.attach_output_handler(
    evaluator,
    event_name=Events.EPOCH_COMPLETED,
    tag="validation",
    metric_names=["nll", "accuracy"],
    global_step_transform=global_step_from_engine(trainer)),
)

# Attach the logger to the trainer to log optimizer's parameters, e.g. learning rate at each iteration
npt_logger.attach_opt_params_handler(
    trainer,
    event_name=Events.ITERATION_STARTED,
    optimizer=optimizer,
    param_name='lr'  # optional
)

# Attach the logger to the trainer to log model's weights norm after each iteration
npt_logger.attach(
    trainer,
    event_name=Events.ITERATION_COMPLETED,
    log_handler=WeightsScalarHandler(model)
)

Explore an experiment with neptune tracking here: https://ui.neptune.ai/o/shared/org/pytorch-ignite-integration/e/PYTOR1-18/charts You can save model checkpoints to a Neptune server:

from ignite.handlers import Checkpoint

def score_function(engine):
    return engine.state.metrics["accuracy"]

to_save = {"model": model}
handler = Checkpoint(
    to_save,
    NeptuneSaver(npt_logger), n_saved=2,
    filename_prefix="best",
    score_function=score_function,
    score_name="validation_accuracy",
    global_step_transform=global_step_from_engine(trainer)
)
validation_evaluator.add_event_handler(Events.COMPLETED, handler)

It is also possible to use the logger as context manager:

from ignite.contrib.handlers.neptune_logger import *

# We are using the api_token for the anonymous user neptuner but you can use your own.

with NeptuneLogger(api_token="ANONYMOUS",
                   project_name="shared/pytorch-ignite-integration",
                   experiment_name="cnn-mnist", # Optional,
                   params={"max_epochs": 10}, # Optional,
                   tags=["pytorch-ignite","mnist"] # Optional
                   ) as npt_logger:

    trainer = Engine(update_fn)
    # Attach the logger to the trainer to log training loss at each iteration
    npt_logger.attach_output_handler(
        trainer,
        event_name=Events.ITERATION_COMPLETED,
        tag="training",
        output_transform=lambda loss: {"loss": loss}
    )

class ignite.contrib.handlers.neptune_logger.NeptuneSaver(neptune_logger)[source]#

Handler that saves input checkpoint to the Neptune server.

Parameters: neptune_logger (NeptuneLogger) – an instance of NeptuneLogger class.

Examples

from ignite.contrib.handlers.neptune_logger import *

# Create a logger
# We are using the api_token for the anonymous user neptuner but you can use your own.

npt_logger = NeptuneLogger(
    api_token="ANONYMOUS",
    project_name="shared/pytorch-ignite-integration",
    experiment_name="cnn-mnist", # Optional,
    params={"max_epochs": 10}, # Optional,
    tags=["pytorch-ignite","minst"] # Optional
)

...
evaluator = create_supervised_evaluator(model, metrics=metrics, ...)
...

from ignite.handlers import Checkpoint

def score_function(engine):
    return engine.state.metrics["accuracy"]

to_save = {"model": model}

# pass neptune logger to NeptuneServer

handler = Checkpoint(
    to_save,
    NeptuneSaver(npt_logger), n_saved=2,
    filename_prefix="best", score_function=score_function,
    score_name="validation_accuracy",
    global_step_transform=global_step_from_engine(trainer)
)

evaluator.add_event_handler(Events.COMPLETED, handler)

# We need to close the logger when we are done
npt_logger.close()

For example, you can access model checkpoints and download them from here: https://ui.neptune.ai/o/shared/org/pytorch-ignite-integration/e/PYTOR1-18/charts

remove(filename)[source]#

Method to remove saved checkpoint.

Parameters: filename (str) – filename associated with checkpoint.
Return type: None

class ignite.contrib.handlers.neptune_logger.OptimizerParamsHandler(optimizer, param_name='lr', tag=None)[source]#

Helper handler to log optimizer parameters

Parameters

optimizer (Optimizer) – torch optimizer or any object with attribute param_groups as a sequence.
param_name (str) – parameter name
tag (Optional[str]) – common title for all produced plots. For example, “generator”

Examples

from ignite.contrib.handlers.neptune_logger import *

# Create a logger
# We are using the api_token for the anonymous user neptuner but you can use your own.

npt_logger = NeptuneLogger(
    api_token="ANONYMOUS",
    project_name="shared/pytorch-ignite-integration",
    experiment_name="cnn-mnist", # Optional,
    params={"max_epochs": 10}, # Optional,
    tags=["pytorch-ignite","minst"] # Optional
)

# Attach the logger to the trainer to log optimizer's parameters, e.g. learning rate at each iteration
npt_logger.attach(
    trainer,
    log_handler=OptimizerParamsHandler(optimizer),
    event_name=Events.ITERATION_STARTED
)
# or equivalently
npt_logger.attach_opt_params_handler(
    trainer,
    event_name=Events.ITERATION_STARTED,
    optimizer=optimizer
)

class ignite.contrib.handlers.neptune_logger.OutputHandler(tag, metric_names=None, output_transform=None, global_step_transform=None, state_attributes=None)[source]#

Helper handler to log engine’s output and/or metrics

Parameters

tag (str) – common title for all produced plots. For example, “training”
metric_names (Optional[Union[str, List[str]]]) – list of metric names to plot or a string “all” to plot all available metrics.
output_transform (Optional[Callable]) – output transform function to prepare engine.state.output as a number. For example, output_transform = lambda output: output This function can also return a dictionary, e.g {“loss”: loss1, “another_loss”: loss2} to label the plot with corresponding keys.
global_step_transform (Optional[Callable[[Engine, Union[str, Events]], int]]) – global step transform function to output a desired global step. Input of the function is (engine, event_name). Output of function should be an integer. Default is None, global_step based on attached engine. If provided, uses function output as global_step. To setup global step from another engine, please use global_step_from_engine().
state_attributes (Optional[List[str]]) – list of attributes of the trainer.state to plot.

Examples

from ignite.contrib.handlers.neptune_logger import *

# Create a logger
# We are using the api_token for the anonymous user neptuner but you can use your own.

npt_logger = NeptuneLogger(
    api_token="ANONYMOUS",
    project_name="shared/pytorch-ignite-integration",
    experiment_name="cnn-mnist", # Optional,
    params={"max_epochs": 10}, # Optional,
    tags=["pytorch-ignite","minst"] # Optional
)

# Attach the logger to the evaluator on the validation dataset and log NLL, Accuracy metrics after
# each epoch. We setup `global_step_transform=global_step_from_engine(trainer)` to take the epoch
# of the `trainer`:
npt_logger.attach(
    evaluator,
    log_handler=OutputHandler(
        tag="validation",
        metric_names=["nll", "accuracy"],
        global_step_transform=global_step_from_engine(trainer)
    ),
    event_name=Events.EPOCH_COMPLETED
)
# or equivalently
npt_logger.attach_output_handler(
    evaluator,
    event_name=Events.EPOCH_COMPLETED,
    tag="validation",
    metric_names=["nll", "accuracy"],
    global_step_transform=global_step_from_engine(trainer)
)

Another example, where model is evaluated every 500 iterations:

from ignite.contrib.handlers.neptune_logger import *

@trainer.on(Events.ITERATION_COMPLETED(every=500))
def evaluate(engine):
    evaluator.run(validation_set, max_epochs=1)

# We are using the api_token for the anonymous user neptuner but you can use your own.

npt_logger = NeptuneLogger(
    api_token="ANONYMOUS",
    project_name="shared/pytorch-ignite-integration",
    experiment_name="cnn-mnist", # Optional,
    params={"max_epochs": 10}, # Optional,
    tags=["pytorch-ignite", "minst"] # Optional
)

def global_step_transform(*args, **kwargs):
    return trainer.state.iteration

# Attach the logger to the evaluator on the validation dataset and log NLL, Accuracy metrics after
# every 500 iterations. Since evaluator engine does not have access to the training iteration, we
# provide a global_step_transform to return the trainer.state.iteration for the global_step, each time
# evaluator metrics are plotted on NeptuneML.

npt_logger.attach_output_handler(
    evaluator,
    event_name=Events.EPOCH_COMPLETED,
    tag="validation",
    metrics=["nll", "accuracy"],
    global_step_transform=global_step_transform
)

Another example where the State Attributes trainer.state.alpha and trainer.state.beta are also logged along with the NLL and Accuracy after each iteration:

npt_logger.attach_output_handler(
    trainer,
    event_name=Events.ITERATION_COMPLETED,
    tag="training",
    metrics=["nll", "accuracy"],
    state_attributes=["alpha", "beta"],
)

Example of global_step_transform:

def global_step_transform(engine, event_name):
    return engine.state.get_event_attrib_value(event_name)

Changed in version 0.4.7: accepts an optional list of state_attributes

class ignite.contrib.handlers.neptune_logger.WeightsScalarHandler(model, reduction=<function norm>, tag=None, whitelist=None)[source]#

Helper handler to log model’s weights as scalars. Handler, upon construction, iterates over named parameters of the model and keep reference to ones permitted by whitelist. Then at every call, applies reduction function to each parameter, produces a scalar and logs it.

Parameters

model (Module) – model to log weights
reduction (Callable[[Tensor], Union[float, Tensor]]) – function to reduce parameters into scalar
tag (Optional[str]) – common title for all produced plots. For example, “generator”
whitelist (Optional[Union[List[str], Callable[[str, Parameter], bool]]]) –
specific weights to log. Should be list of model’s submodules or parameters names, or a callable which gets weight along with its name and determines if it should be logged. Names should be fully-qualified. For more information please refer to PyTorch docs. If not given, all of model’s weights are logged.

Examples

from ignite.contrib.handlers.neptune_logger import *

# Create a logger
# We are using the api_token for the anonymous user neptuner but you can use your own.

npt_logger = NeptuneLogger(
    api_token="ANONYMOUS",
    project_name="shared/pytorch-ignite-integration",
    experiment_name="cnn-mnist", # Optional,
    params={"max_epochs": 10}, # Optional,
    tags=["pytorch-ignite","minst"] # Optional
)

# Attach the logger to the trainer to log model's weights norm after each iteration
npt_logger.attach(
    trainer,
    event_name=Events.ITERATION_COMPLETED,
    log_handler=WeightsScalarHandler(model, reduction=torch.norm)
)

from ignite.contrib.handlers.neptune_logger import *

npt_logger = NeptuneLogger(
    api_token="ANONYMOUS",
    project_name="shared/pytorch-ignite-integration",
    experiment_name="cnn-mnist", # Optional,
    params={"max_epochs": 10}, # Optional,
    tags=["pytorch-ignite","minst"] # Optional
)

# Log only `fc` weights
npt_logger.attach(
    trainer,
    event_name=Events.ITERATION_COMPLETED,
    log_handler=WeightsScalarHandler(
        model,
        whitelist=['fc']
    )
)

from ignite.contrib.handlers.neptune_logger import *

npt_logger = NeptuneLogger(
    api_token="ANONYMOUS",
    project_name="shared/pytorch-ignite-integration",
    experiment_name="cnn-mnist", # Optional,
    params={"max_epochs": 10}, # Optional,
    tags=["pytorch-ignite","minst"] # Optional
)

# Log weights which have `bias` in their names
def has_bias_in_name(n, p):
    return 'bias' in n

npt_logger.attach(
    trainer,
    event_name=Events.ITERATION_COMPLETED,
    log_handler=WeightsScalarHandler(model, whitelist=has_bias_in_name)
)

Changed in version 0.4.9: optional argument whitelist added.

ignite.contrib.handlers.neptune_logger.global_step_from_engine(engine, custom_event_name=None)[source]#

Helper method to setup global_step_transform function using another engine. This can be helpful for logging trainer epoch/iteration while output handler is attached to an evaluator.

Parameters

engine (Engine) – engine which state is used to provide the global step
custom_event_name (Optional[Events]) – registered event name. Optional argument, event name to use.

Returns

global step based on provided engine

Return type

Callable