FAQ
===
In this section we grouped answers on frequently asked questions and some best practices of using `ignite`.
Each engine has its own Events
------------------------------
It is important to understand that engines have their own events. For example, we defined a trainer and an evaluator:
.. code-block:: python
@trainer.on(Events.EPOCH_COMPLETED)
def in_training_loop_on_epoch_completed(engine):
evaluator.run(val_loader) # this starts another loop on validation dataset to compute metrics
@evaluator.on(Events.COMPLETED)
def when_validation_loop_is_done(engine):
# do something with computed metrics etc
# -> early stopping or reduce LR on plateau
# or just log them
Trainer engine has its own loop and runs multiple times over the training dataset. When a training epoch is over we
launch evaluator engine and run a single time of over the validation dataset. **Evaluator has its own loop**. Therefore,
it runs only one epoch and `Events.EPOCH_COMPLETED` is equivalent to `Events.COMPLETED`.
As a consequence, the following code is correct too:
.. code-block:: python
handler = EarlyStopping(patience=10, score_function=score_function, trainer=trainer)
evaluator.add_event_handler(Events.COMPLETED, handler)
best_model_saver = ModelCheckpoint('/tmp/models', 'best', score_function=score_function)
evaluator.add_event_handler(Events.COMPLETED, best_model_saver, {'mymodel': model})
More details :ref:`Events and Handlers:`.
Creating Custom Events based on Forward/Backward Pass
-----------------------------------------------------
There are cases where the user might want to add events based on the loss calculation and backward pass. Ignite provides
flexibility to the user to allow for this:
.. code-block:: python
from ignite.engine import EventEnum
class BackpropEvents(EventEnum):
"""
Events based on back propagation
"""
BACKWARD_STARTED = 'backward_started'
BACKWARD_COMPLETED = 'backward_completed'
OPTIM_STEP_COMPLETED = 'optim_step_completed'
def update(engine, batch):
model.train()
opitmizer.zero_grad()
x, y = process_batch(batch)
y_pred = model(x)
loss = loss_fn(y_pred, y)
engine.fire_event(BackpropEvents.BACKWARD_STARTED)
loss.backward()
engine.fire_event(BackpropEvents.BACKWARD_COMPLETED)
optimizer.step()
engine.fire_event(BackpropEvents.OPTIM_STEP_COMPLETED)
return loss.item()
trainer = Engine(update)
trainer.register_events(*BackpropEvents)
@trainer.on(BackpropEvents.BACKWARD_STARTED)
def function_before_backprop(engine):
# insert custom function here
.. Note ::
Events defined by user should inherit from :class:`~ignite.engine.events.EventEnum`
More detailed implementation can be found in `TBPTT Trainer <https://pytorch.org/ignite/master/_modules/ignite/contrib/engines/tbptt.html#create_supervised_tbptt_trainer>`_.
Gradients accumulation
----------------------
A best practice to use if we need to increase effectively the batch size on limited GPU resources. There several ways to
do this, the most simple is the following:
.. code-block:: python
accumulation_steps = 4
def update_fn(engine, batch):
model.train()
x, y = prepare_batch(batch, device=device, non_blocking=non_blocking)
y_pred = model(x)
loss = criterion(y_pred, y) / accumulation_steps
loss.backward()
if engine.state.iteration % accumulation_steps == 0:
optimizer.step()
optimizer.zero_grad()
return loss.item()
trainer = Engine(update_fn)
Based on `this blog article <https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255>`_ and
`this code <https://gist.github.com/thomwolf/ac7a7da6b1888c2eeac8ac8b9b05d3d3#file-gradient_accumulation-py>`_.
Working with iterators
----------------------
If data provider for training or validation is an iterator (infinite or finite with known or unknown size), here are
basic examples of how to setup trainer or evaluator.
Infinite iterator for training
``````````````````````````````
Let's use an infinite data iterator as training dataflow
.. code-block:: python
import torch
from ignite.engine import Engine, Events
torch.manual_seed(12)
def infinite_iterator(batch_size):
while True:
batch = torch.rand(batch_size, 3, 32, 32)
yield batch
def train_step(trainer, batch):
# ...
s = trainer.state
print(
f"{s.epoch}/{s.max_epochs} : {s.iteration} - {batch.norm():.3f}"
)
trainer = Engine(train_step)
# We need to specify epoch_length to define the epoch
trainer.run(infinite_iterator(4), epoch_length=5, max_epochs=3)
In this case we will obtain the following output:
.. code-block:: text
1/3 : 1 - 63.862
1/3 : 2 - 64.042
1/3 : 3 - 63.936
1/3 : 4 - 64.141
1/3 : 5 - 64.767
2/3 : 6 - 63.791
2/3 : 7 - 64.565
2/3 : 8 - 63.602
2/3 : 9 - 63.995
2/3 : 10 - 63.943
3/3 : 11 - 63.831
3/3 : 12 - 64.276
3/3 : 13 - 64.148
3/3 : 14 - 63.920
3/3 : 15 - 64.226
If we do not specify `epoch_length`, we can stop the training explicitly by calling :meth:`~ignite.engine.engine.Engine.terminate`
In this case, there will be only a single epoch defined.
.. code-block:: python
import torch
from ignite.engine import Engine, Events
torch.manual_seed(12)
def infinite_iterator(batch_size):
while True:
batch = torch.rand(batch_size, 3, 32, 32)
yield batch
def train_step(trainer, batch):
# ...
s = trainer.state
print(
f"{s.epoch}/{s.max_epochs} : {s.iteration} - {batch.norm():.3f}"
)
trainer = Engine(train_step)
@trainer.on(Events.ITERATION_COMPLETED(once=15))
def stop_training():
trainer.terminate()
trainer.run(infinite_iterator(4))
We obtain the following output:
.. code-block:: text
1/1 : 1 - 63.862
1/1 : 2 - 64.042
1/1 : 3 - 63.936
1/1 : 4 - 64.141
1/1 : 5 - 64.767
1/1 : 6 - 63.791
1/1 : 7 - 64.565
1/1 : 8 - 63.602
1/1 : 9 - 63.995
1/1 : 10 - 63.943
1/1 : 11 - 63.831
1/1 : 12 - 64.276
1/1 : 13 - 64.148
1/1 : 14 - 63.920
1/1 : 15 - 64.226
Same code can be used for validating models.
Finite iterator with unknown length
```````````````````````````````````
Let's use a finite data iterator but with unknown length (for user). In case of training, we would like to perform
several passes over the dataflow and thus we need to restart the data iterator when it is exhausted.
In the code, we do not specify `epoch_length` which will be automatically determined.
.. code-block:: python
import torch
from ignite.engine import Engine, Events
torch.manual_seed(12)
def finite_unk_size_data_iter():
for i in range(11):
yield i
def train_step(trainer, batch):
# ...
s = trainer.state
print(
f"{s.epoch}/{s.max_epochs} : {s.iteration} - {batch:.3f}"
)
trainer = Engine(train_step)
@trainer.on(Events.DATALOADER_STOP_ITERATION)
def restart_iter():
trainer.state.dataloader = finite_unk_size_data_iter()
data_iter = finite_unk_size_data_iter()
trainer.run(data_iter, max_epochs=5)
In case of validation, the code is simply
.. code-block:: python
import torch
from ignite.engine import Engine, Events
torch.manual_seed(12)
def finite_unk_size_data_iter():
for i in range(11):
yield i
def val_step(evaluator, batch):
# ...
s = evaluator.state
print(
f"{s.epoch}/{s.max_epochs} : {s.iteration} - {batch:.3f}"
)
evaluator = Engine(val_step)
data_iter = finite_unk_size_data_iter()
evaluator.run(data_iter)
Finite iterator with known length
`````````````````````````````````
Let's use a finite data iterator with known size for training or validation.
If we need to restart the data iterator, we can do this either as in case of
unknown size by attaching the restart handler on `@trainer.on(Events.DATALOADER_STOP_ITERATION)`,
but here we will do this explicitly on iteration:
.. code-block:: python
import torch
from ignite.engine import Engine, Events
torch.manual_seed(12)
size = 11
def finite_size_data_iter(size):
for i in range(size):
yield i
def train_step(trainer, batch):
# ...
s = trainer.state
print(
f"{s.epoch}/{s.max_epochs} : {s.iteration} - {batch:.3f}"
)
trainer = Engine(train_step)
@trainer.on(Events.ITERATION_COMPLETED(every=size))
def restart_iter():
trainer.state.dataloader = finite_size_data_iter(size)
data_iter = finite_size_data_iter(size)
trainer.run(data_iter, max_epochs=5)
In case of validation, the code is simply
.. code-block:: python
import torch
from ignite.engine import Engine, Events
torch.manual_seed(12)
size = 11
def finite_size_data_iter(size):
for i in range(size):
yield i
def val_step(evaluator, batch):
# ...
s = evaluator.state
print(
f"{s.epoch}/{s.max_epochs} : {s.iteration} - {batch:.3f}"
)
evaluator = Engine(val_step)
data_iter = finite_size_data_iter(size)
evaluator.run(data_iter)
Switching data provider during the training
-------------------------------------------
User can easily switch data provider during the training using :meth:`~ignite.engine.engine.Engine.set_data`.
See an example in the documentation of the method.
Time profiling during training
------------------------------
User can fetch times in several manners depending on complexity of required time profiling:
Single epoch and total time
```````````````````````````
Simpliest way to fetch time of single epoch and complete training is to use
``engine.state.times["EPOCH_COMPLETED"]`` and ``engine.state.times["COMPLETED"]``:
.. code-block:: python
trainer = ...
@trainer.on(Events.EPOCH_COMPLETED)
def log_epoch_time():
print(f"{trainer.state.epoch}: {trainer.state.times['EPOCH_COMPLETED']}")
@trainer.on(Events.COMPLETED)
def log_total_time():
print(f"Total: {trainer.state.times['COMPLETED']}")
For details, see :class:`~ignite.engine.events.State`.
Basic time profiling
````````````````````
User can setup :class:`~ignite.handlers.time_profilers.BasicTimeProfiler` to fetch times spent in data
processing, training step, event handlers:
.. code-block:: python
from ignite.handlers import BasicTimeProfiler
trainer = ...
# Create an object of the profiler and attach an engine to it
profiler = BasicTimeProfiler()
profiler.attach(trainer)
@trainer.on(Events.EPOCH_COMPLETED(every=10))
def log_intermediate_results():
profiler.print_results(profiler.get_results())
trainer.run(dataloader, max_epochs=3)
Typical output:
.. code-block:: text
----------------------------------------------------
| Time profiling stats (in seconds): |
----------------------------------------------------
total | min/index | max/index | mean | std
Processing function:
157.46292 | 0.01452/1501 | 0.26905/0 | 0.07730 | 0.01258
Dataflow:
6.11384 | 0.00008/1935 | 0.28461/1551 | 0.00300 | 0.02693
Event handlers:
2.82721
- Events.STARTED: []
0.00000
- Events.EPOCH_STARTED: []
0.00006 | 0.00000/0 | 0.00000/17 | 0.00000 | 0.00000
- Events.ITERATION_STARTED: ['PiecewiseLinear']
0.03482 | 0.00001/188 | 0.00018/679 | 0.00002 | 0.00001
- Events.ITERATION_COMPLETED: ['TerminateOnNan']
0.20037 | 0.00006/866 | 0.00089/1943 | 0.00010 | 0.00003
- Events.EPOCH_COMPLETED: ['empty_cuda_cache', 'training.<locals>.log_elapsed_time', ]
2.57860 | 0.11529/0 | 0.14977/13 | 0.12893 | 0.00790
- Events.COMPLETED: []
not yet triggered
For details, see :class:`~ignite.handlers.time_profilers.BasicTimeProfiler`.
Event handlers time profiling
`````````````````````````````
If you want to get time breakdown per handler basis then you can setup
:class:`~ignite.handlers.time_profilers.HandlersTimeProfiler`:
.. code-block:: python
from ignite.handlers import HandlersTimeProfiler
trainer = ...
# Create an object of the profiler and attach an engine to it
profiler = HandlersTimeProfiler()
profiler.attach(trainer)
@trainer.on(Events.EPOCH_COMPLETED(every=10))
def log_intermediate_results():
profiler.print_results(profiler.get_results())
trainer.run(dataloader, max_epochs=3)
Typical output:
.. code-block:: text
----------------------------------------- ----------------------- -------------- ...
Handler Event Name Total(s)
----------------------------------------- ----------------------- --------------
run.<locals>.log_training_results EPOCH_COMPLETED 19.43245
run.<locals>.log_validation_results EPOCH_COMPLETED 2.55271
run.<locals>.log_time EPOCH_COMPLETED 0.00049
run.<locals>.log_intermediate_results EPOCH_COMPLETED 0.00106
run.<locals>.log_training_loss ITERATION_COMPLETED 0.059
run.<locals>.log_time COMPLETED not triggered
----------------------------------------- ----------------------- --------------
Total 22.04571
----------------------------------------- ----------------------- --------------
Processing took total 11.29543s [min/index: 0.00393s/1875, max/index: 0.00784s/0,
mean: 0.00602s, std: 0.00034s]
Dataflow took total 16.24365s [min/index: 0.00533s/1874, max/index: 0.01129s/937,
mean: 0.00866s, std: 0.00113s]
For details, see :class:`~ignite.handlers.time_profilers.HandlersTimeProfiler`.
Custom time measures
````````````````````
Custom time measures can be performed using :class:`~ignite.handlers.timing.Timer`. See its docstring for details.
Other questions
---------------
Other questions and answers can be also found on the github among the issues labeled by
`question <https://github.com/pytorch/ignite/issues?utf8=%E2%9C%93&q=is%3Aissue+label%3Aquestion+>`_ and on the forum
`Discuss.PyTorch <https://discuss.pytorch.org/c/ignite/15>`_, category "Ignite".