Shortcuts

FAQ#

In this section we grouped answers on frequently asked questions and some best practices of using ignite.

Each engine has its own Events#

It is important to understand that engines have their own events. For example, we defined a trainer and an evaluator:

@trainer.on(Events.EPOCH_COMPLETED)
def in_training_loop_on_epoch_completed(engine):
    evaluator.run(val_loader) # this starts another loop on validation dataset to compute metrics

@evaluator.on(Events.COMPLETED)
def when_validation_loop_is_done(engine):
    # do something with computed metrics etc
    # -> early stopping or reduce LR on plateau
    # or just log them

Trainer engine has its own loop and runs multiple times over the training dataset. When a training epoch is over we launch evaluator engine and run a single time of over the validation dataset. Evaluator has its own loop. Therefore, it runs only one epoch and EPOCH_COMPLETED is equivalent to COMPLETED. As a consequence, the following code is correct too:

handler = EarlyStopping(patience=10, score_function=score_function, trainer=trainer)
evaluator.add_event_handler(Events.COMPLETED, handler)

best_model_saver = ModelCheckpoint('/tmp/models', 'best', score_function=score_function)
evaluator.add_event_handler(Events.COMPLETED, best_model_saver, {'mymodel': model})

More details Events and Handlers.

Creating Custom Events based on Forward/Backward Pass#

There are cases where the user might want to add events based on the loss calculation and backward pass. Ignite provides flexibility to the user to allow for this:

class BackpropEvents(Enum):
    """
    Events based on back propagation
    """
    BACKWARD_STARTED = 'backward_started'
    BACKWARD_COMPLETED = 'backward_completed'
    OPTIM_STEP_COMPLETED = 'optim_step_completed'

def update(engine, batch):
    model.train()
    opitmizer.zero_grad()
    x, y = process_batch(batch)
    y_pred = model(x)
    loss = loss_fn(y_pred, y)
    engine.fire_event(BackpropEvents.BACKWARD_STARTED)
    loss.backward()
    engine.fire_event(BackpropEvents.BACKWARD_COMPLETED)
    optimizer.step()
    engine.fire_event(BackpropEvents.OPTIM_STEP_COMPLETED)

    return loss.item()

trainer = Engine(update)
trainer.register_events(*BackpropEvents)

@trainer.on(BackpropEvents.BACKWARD_STARTED)
def function_before_backprop(engine):
    # insert custom function here

More detailed implementation can be found in TBPTT Trainer.

Creating Custom Events based on Iteration and Epoch#

Another type of custom event could be based on number of iteration and epochs. Ignite has CustomPeriodicEvent, which allows the user to define events based on number of elapsed iterations/epochs.

Gradients accumulation#

A best practice to use if we need to increase effectively the batchsize on limited GPU resources. There several ways to do this, the most simple is the following:

accumulation_steps = 4

def update_fn(engine, batch):
    model.train()

    x, y = prepare_batch(batch, device=device, non_blocking=non_blocking)
    y_pred = model(x)
    loss = criterion(y_pred, y) / accumulation_steps
    loss.backward()

    if engine.state.iteration % accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()

    return loss.item()

trainer = Engine(update_fn)

Based on this blog article and this code.

Other answers can be found on the github among the issues labeled by question.