ignite.contrib.metrics#
Contribution module of metrics
Computes Average Precision accumulating predictions and the groundtruth during an epoch and applying sklearn.metrics.average_precision_score . 

Provides GPU information: a) used memory percentage, b) gpu utilization percentage values as Metric on each iterations. 

Compute precisionrecall pairs for different probability thresholds for binary classification task by accumulating predictions and the groundtruth during an epoch and applying sklearn.metrics.precision_recall_curve . 

Computes Area Under the Receiver Operating Characteristic Curve (ROC AUC) accumulating predictions and the groundtruth during an epoch and applying sklearn.metrics.roc_auc_score . 

Compute Receiver operating characteristic (ROC) for binary classification task by accumulating predictions and the groundtruth during an epoch and applying sklearn.metrics.roc_curve . 
 class ignite.contrib.metrics.AveragePrecision(output_transform=<function AveragePrecision.<lambda>>, check_compute_fn=False)[source]#
Computes Average Precision accumulating predictions and the groundtruth during an epoch and applying sklearn.metrics.average_precision_score .
 Parameters
output_transform (Callable) – a callable that is used to transform the
Engine
’sprocess_function
’s output into the form expected by the metric. This can be useful if, for example, you have a multioutput model and you want to compute the metric with respect to one of the outputs.check_compute_fn (bool) – Default False. If True, average_precision_score is run on the first batch of data to ensure there are no issues. User will be warned in case there are any issues computing the function.
AveragePrecision expects y to be comprised of 0’s and 1’s. y_pred must either be probability estimates or confidence values. To apply an activation to y_pred, use output_transform as shown below:
def activated_output_transform(output): y_pred, y = output y_pred = torch.softmax(y_pred, dim=1) return y_pred, y avg_precision = AveragePrecision(activated_output_transform)
 class ignite.contrib.metrics.GpuInfo[source]#
Provides GPU information: a) used memory percentage, b) gpu utilization percentage values as Metric on each iterations.
Note
In case if gpu utilization reports “N/A” on a given GPU, corresponding metric value is not set.
Examples
# Default GPU measurements GpuInfo().attach(trainer, name='gpu') # metric names are 'gpu:X mem(%)', 'gpu:X util(%)' # Logging with TQDM ProgressBar(persist=True).attach(trainer, metric_names=['gpu:0 mem(%)', 'gpu:0 util(%)']) # Progress bar will looks like # Epoch [2/10]: [12/24] 50%█████ , gpu:0 mem(%)=79, gpu:0 util(%)=59 [00:17<1:23] # Logging with Tensorboard tb_logger.attach(trainer, log_handler=OutputHandler(tag="training", metric_names='all'), event_name=Events.ITERATION_COMPLETED)
 attach(engine, name='gpu', event_name=Events.ITERATION_COMPLETED)[source]#
Attaches current metric to provided engine. On the end of engine’s run, engine.state.metrics dictionary will contain computed metric’s value under provided name.
 Parameters
engine (Engine) – the engine to which the metric must be attached
name (str) – the name of the metric to attach
usage – the usage of the metric. Valid string values should be
ignite.metrics.EpochWise.usage_name
(default) orignite.metrics.BatchWise.usage_name
.
 Return type
None
Example:
metric = ... metric.attach(engine, "mymetric") assert "mymetric" in engine.run(data).metrics assert metric.is_attached(engine)
Example with usage:
metric = ... metric.attach(engine, "mymetric", usage=BatchWise.usage_name) assert "mymetric" in engine.run(data).metrics assert metric.is_attached(engine, usage=BatchWise.usage_name)
 completed(engine, name)[source]#
Helper method to compute metric’s value and put into the engine. It is automatically attached to the engine with
attach()
. Parameters
 Return type
None
Changed in version 0.4.3: Added dict in metrics results.
 compute()[source]#
Computes the metric based on it’s accumulated state.
By default, this is called at the end of each epoch.
 Returns
 the actual quantity of interest. However, if a
Mapping
is returned, it will be (shallow) flattened into engine.state.metrics whencompleted()
is called.  Return type
Any
 Raises
NotComputableError – raised when the metric cannot be computed.
 class ignite.contrib.metrics.PrecisionRecallCurve(output_transform=<function PrecisionRecallCurve.<lambda>>, check_compute_fn=False)[source]#
Compute precisionrecall pairs for different probability thresholds for binary classification task by accumulating predictions and the groundtruth during an epoch and applying sklearn.metrics.precision_recall_curve .
 Parameters
output_transform (Callable) – a callable that is used to transform the
Engine
’sprocess_function
’s output into the form expected by the metric. This can be useful if, for example, you have a multioutput model and you want to compute the metric with respect to one of the outputs.check_compute_fn (bool) – Default False. If True, precision_recall_curve is run on the first batch of data to ensure there are no issues. User will be warned in case there are any issues computing the function.
PrecisionRecallCurve expects y to be comprised of 0’s and 1’s. y_pred must either be probability estimates or confidence values. To apply an activation to y_pred, use output_transform as shown below:
def activated_output_transform(output): y_pred, y = output y_pred = torch.sigmoid(y_pred) return y_pred, y roc_auc = PrecisionRecallCurve(activated_output_transform)
 class ignite.contrib.metrics.ROC_AUC(output_transform=<function ROC_AUC.<lambda>>, check_compute_fn=False)[source]#
Computes Area Under the Receiver Operating Characteristic Curve (ROC AUC) accumulating predictions and the groundtruth during an epoch and applying sklearn.metrics.roc_auc_score .
 Parameters
output_transform (Callable) – a callable that is used to transform the
Engine
’sprocess_function
’s output into the form expected by the metric. This can be useful if, for example, you have a multioutput model and you want to compute the metric with respect to one of the outputs.check_compute_fn (bool) – Default False. If True, roc_curve is run on the first batch of data to ensure there are no issues. User will be warned in case there are any issues computing the function.
ROC_AUC expects y to be comprised of 0’s and 1’s. y_pred must either be probability estimates or confidence values. To apply an activation to y_pred, use output_transform as shown below:
def activated_output_transform(output): y_pred, y = output y_pred = torch.sigmoid(y_pred) return y_pred, y roc_auc = ROC_AUC(activated_output_transform)
 class ignite.contrib.metrics.RocCurve(output_transform=<function RocCurve.<lambda>>, check_compute_fn=False)[source]#
Compute Receiver operating characteristic (ROC) for binary classification task by accumulating predictions and the groundtruth during an epoch and applying sklearn.metrics.roc_curve .
 Parameters
output_transform (Callable) – a callable that is used to transform the
Engine
’sprocess_function
’s output into the form expected by the metric. This can be useful if, for example, you have a multioutput model and you want to compute the metric with respect to one of the outputs.check_compute_fn (bool) –
Default False. If True, sklearn.metrics.roc_curve is run on the first batch of data to ensure there are no issues. User will be warned in case there are any issues computing the function.
RocCurve expects y to be comprised of 0’s and 1’s. y_pred must either be probability estimates or confidence values. To apply an activation to y_pred, use output_transform as shown below:
def activated_output_transform(output): y_pred, y = output y_pred = torch.sigmoid(y_pred) return y_pred, y roc_auc = RocCurve(activated_output_transform)
Regression metrics#
Module ignite.contrib.metrics.regression
provides implementations of
metrics useful for regression tasks. Definitions of metrics are based on Botchkarev 2018, page 30 “Appendix 2. Metrics mathematical definitions”.
Complete list of metrics:
Calculates the Canberra Metric. 

Calculates the Fractional Absolute Error. 

Calculates the Fractional Bias. 

Calculates the Geometric Mean Absolute Error. 

Calculates the Geometric Mean Relative Absolute Error. 

Calculates the Manhattan Distance. 

Calculates the Maximum Absolute Error. 

Calculate Mean Absolute Relative Error. 

Calculates the Mean Error. 

Calculates the Mean Normalized Bias. 

Calculates the Median Absolute Error. 

Calculates the Median Absolute Percentage Error. 

Calculates the Median Relative Absolute Error. 

Calculates the RSquared, the coefficient of determination. 

Calculates the Wave Hedges Distance. 
 class ignite.contrib.metrics.regression.CanberraMetric(output_transform=<function Metric.<lambda>>, device=device(type='cpu'))[source]#
Calculates the Canberra Metric.
$\text{CM} = \sum_{j=1}^n\frac{A_j  P_j}{A_j + P_j}$where, $A_j$ is the ground truth and $P_j$ is the predicted value.
More details can be found in Botchkarev 2018 or scikitlearn distance metrics
update
must receive output of the form(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.y and y_pred must be of same shape (N, ) or (N, 1).
Parameters are inherited from
Metric.__init__
. Parameters
output_transform (Callable) – a callable that is used to transform the
Engine
’sprocess_function
’s output into the form expected by the metric. This can be useful if, for example, you have a multioutput model and you want to compute the metric with respect to one of the outputs. By default, metrics require the output as(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.device (Union[str, device]) – specifies which device updates are accumulated on. Setting the metric’s device to be the same as your
update
arguments ensures theupdate
method is nonblocking. By default, CPU.
Changed in version 0.4.3:
Fixed implementation:
abs
in denominator.Works with DDP.
 class ignite.contrib.metrics.regression.FractionalAbsoluteError(output_transform=<function Metric.<lambda>>, device=device(type='cpu'))[source]#
Calculates the Fractional Absolute Error.
$\text{FAE} = \frac{1}{n}\sum_{j=1}^n\frac{2 A_j  P_j}{A_j + P_j}$where, $A_j$ is the ground truth and $P_j$ is the predicted value.
More details can be found in Botchkarev 2018.
update
must receive output of the form(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.y and y_pred must be of same shape (N, ) or (N, 1).
Parameters are inherited from
Metric.__init__
. Parameters
output_transform (Callable) – a callable that is used to transform the
Engine
’sprocess_function
’s output into the form expected by the metric. This can be useful if, for example, you have a multioutput model and you want to compute the metric with respect to one of the outputs. By default, metrics require the output as(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.device (Union[str, device]) – specifies which device updates are accumulated on. Setting the metric’s device to be the same as your
update
arguments ensures theupdate
method is nonblocking. By default, CPU.
 class ignite.contrib.metrics.regression.FractionalBias(output_transform=<function Metric.<lambda>>, device=device(type='cpu'))[source]#
Calculates the Fractional Bias.
$\text{FB} = \frac{1}{n}\sum_{j=1}^n\frac{2 (A_j  P_j)}{A_j + P_j}$where $A_j$ is the ground truth and $P_j$ is the predicted value.
More details can be found in Botchkarev 2018.
update
must receive output of the form(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.y and y_pred must be of same shape (N, ) or (N, 1).
Parameters are inherited from
Metric.__init__
. Parameters
output_transform (Callable) – a callable that is used to transform the
Engine
’sprocess_function
’s output into the form expected by the metric. This can be useful if, for example, you have a multioutput model and you want to compute the metric with respect to one of the outputs. By default, metrics require the output as(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.device (Union[str, device]) – specifies which device updates are accumulated on. Setting the metric’s device to be the same as your
update
arguments ensures theupdate
method is nonblocking. By default, CPU.
 class ignite.contrib.metrics.regression.GeometricMeanAbsoluteError(output_transform=<function Metric.<lambda>>, device=device(type='cpu'))[source]#
Calculates the Geometric Mean Absolute Error.
$\text{GMAE} = \exp(\frac{1}{n}\sum_{j=1}^n\ln(A_j  P_j))$where, $A_j$ is the ground truth and $P_j$ is the predicted value.
More details can be found in Botchkarev 2018.
update
must receive output of the form(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.y and y_pred must be of same shape (N, ) or (N, 1).
Parameters are inherited from
Metric.__init__
. Parameters
output_transform (Callable) – a callable that is used to transform the
Engine
’sprocess_function
’s output into the form expected by the metric. This can be useful if, for example, you have a multioutput model and you want to compute the metric with respect to one of the outputs. By default, metrics require the output as(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.device (Union[str, device]) – specifies which device updates are accumulated on. Setting the metric’s device to be the same as your
update
arguments ensures theupdate
method is nonblocking. By default, CPU.
 class ignite.contrib.metrics.regression.GeometricMeanRelativeAbsoluteError(output_transform=<function Metric.<lambda>>, device=device(type='cpu'))[source]#
Calculates the Geometric Mean Relative Absolute Error.
$\text{GMRAE} = \exp(\frac{1}{n}\sum_{j=1}^n \ln\frac{A_j  P_j}{A_j  \bar{A}})$where $A_j$ is the ground truth and $P_j$ is the predicted value.
More details can be found in Botchkarev 2018.
update
must receive output of the form(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.y and y_pred must be of same shape (N, ) or (N, 1).
Parameters are inherited from
Metric.__init__
. Parameters
output_transform (Callable) – a callable that is used to transform the
Engine
’sprocess_function
’s output into the form expected by the metric. This can be useful if, for example, you have a multioutput model and you want to compute the metric with respect to one of the outputs. By default, metrics require the output as(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.device (Union[str, device]) – specifies which device updates are accumulated on. Setting the metric’s device to be the same as your
update
arguments ensures theupdate
method is nonblocking. By default, CPU.
 class ignite.contrib.metrics.regression.ManhattanDistance(output_transform=<function Metric.<lambda>>, device=device(type='cpu'))[source]#
Calculates the Manhattan Distance.
$\text{MD} = \sum_{j=1}^n A_j  P_j$where $A_j$ is the ground truth and $P_j$ is the predicted value.
More details can be found in scikitlearn distance metrics.
update
must receive output of the form(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.y and y_pred must be of same shape (N, ) or (N, 1).
Parameters are inherited from
Metric.__init__
. Parameters
output_transform (Callable) – a callable that is used to transform the
Engine
’sprocess_function
’s output into the form expected by the metric. This can be useful if, for example, you have a multioutput model and you want to compute the metric with respect to one of the outputs. By default, metrics require the output as(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.device (Union[str, device]) – specifies which device updates are accumulated on. Setting the metric’s device to be the same as your
update
arguments ensures theupdate
method is nonblocking. By default, CPU.
Changed in version 0.4.3:
Fixed sklearn compatibility.
Workes with DDP.
 class ignite.contrib.metrics.regression.MaximumAbsoluteError(output_transform=<function Metric.<lambda>>, device=device(type='cpu'))[source]#
Calculates the Maximum Absolute Error.
$\text{MaxAE} = \max_{j=1,n} \left( \lvert A_jP_j \rvert \right)$where $A_j$ is the ground truth and $P_j$ is the predicted value.
More details can be found in Botchkarev 2018.
update
must receive output of the form(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.y and y_pred must be of same shape (N, ) or (N, 1).
Parameters are inherited from
Metric.__init__
. Parameters
output_transform (Callable) – a callable that is used to transform the
Engine
’sprocess_function
’s output into the form expected by the metric. This can be useful if, for example, you have a multioutput model and you want to compute the metric with respect to one of the outputs. By default, metrics require the output as(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.device (Union[str, device]) – specifies which device updates are accumulated on. Setting the metric’s device to be the same as your
update
arguments ensures theupdate
method is nonblocking. By default, CPU.
 class ignite.contrib.metrics.regression.MeanAbsoluteRelativeError(output_transform=<function Metric.<lambda>>, device=device(type='cpu'))[source]#
Calculate Mean Absolute Relative Error.
$\text{MARE} = \frac{1}{n}\sum_{j=1}^n\frac{\leftA_jP_j\right}{\leftA_j\right}$where $A_j$ is the ground truth and $P_j$ is the predicted value.
More details can be found in the reference Botchkarev 2018.
update
must receive output of the form(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.y and y_pred must be of same shape (N, ) or (N, 1).
Parameters are inherited from
Metric.__init__
. Parameters
output_transform (Callable) – a callable that is used to transform the
Engine
’sprocess_function
’s output into the form expected by the metric. This can be useful if, for example, you have a multioutput model and you want to compute the metric with respect to one of the outputs. By default, metrics require the output as(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.device (Union[str, device]) – specifies which device updates are accumulated on. Setting the metric’s device to be the same as your
update
arguments ensures theupdate
method is nonblocking. By default, CPU.
 class ignite.contrib.metrics.regression.MeanError(output_transform=<function Metric.<lambda>>, device=device(type='cpu'))[source]#
Calculates the Mean Error.
$\text{ME} = \frac{1}{n}\sum_{j=1}^n (A_j  P_j)$where $A_j$ is the ground truth and $P_j$ is the predicted value.
More details can be found in the reference Botchkarev 2018.
update
must receive output of the form(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.y and y_pred must be of same shape (N, ) or (N, 1).
Parameters are inherited from
Metric.__init__
. Parameters
output_transform (Callable) – a callable that is used to transform the
Engine
’sprocess_function
’s output into the form expected by the metric. This can be useful if, for example, you have a multioutput model and you want to compute the metric with respect to one of the outputs. By default, metrics require the output as(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.device (Union[str, device]) – specifies which device updates are accumulated on. Setting the metric’s device to be the same as your
update
arguments ensures theupdate
method is nonblocking. By default, CPU.
 class ignite.contrib.metrics.regression.MeanNormalizedBias(output_transform=<function Metric.<lambda>>, device=device(type='cpu'))[source]#
Calculates the Mean Normalized Bias.
$\text{MNB} = \frac{1}{n}\sum_{j=1}^n\frac{A_j  P_j}{A_j}$where $A_j$ is the ground truth and $P_j$ is the predicted value.
More details can be found in the reference Botchkarev 2018.
update
must receive output of the form(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.y and y_pred must be of same shape (N, ) or (N, 1).
Parameters are inherited from
Metric.__init__
. Parameters
output_transform (Callable) – a callable that is used to transform the
Engine
’sprocess_function
’s output into the form expected by the metric. This can be useful if, for example, you have a multioutput model and you want to compute the metric with respect to one of the outputs. By default, metrics require the output as(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.device (Union[str, device]) – specifies which device updates are accumulated on. Setting the metric’s device to be the same as your
update
arguments ensures theupdate
method is nonblocking. By default, CPU.
 class ignite.contrib.metrics.regression.MedianAbsoluteError(output_transform=<function MedianAbsoluteError.<lambda>>)[source]#
Calculates the Median Absolute Error.
$\text{MdAE} = \text{MD}_{j=1,n} \left( A_j  P_j \right)$where $A_j$ is the ground truth and $P_j$ is the predicted value.
More details can be found in Botchkarev 2018.
update
must receive output of the form(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.y and y_pred must be of same shape (N, ) or (N, 1) and of type float32.
Warning
Current implementation stores all input data (output and target) in as tensors before computing a metric. This can potentially lead to a memory error if the input data is larger than available RAM.
 Parameters
output_transform (Callable) – a callable that is used to transform the
Engine
’sprocess_function
’s output into the form expected by the metric. This can be useful if, for example, you have a multioutput model and you want to compute the metric with respect to one of the outputs. By default, metrics require the output as(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.
 class ignite.contrib.metrics.regression.MedianAbsolutePercentageError(output_transform=<function MedianAbsolutePercentageError.<lambda>>)[source]#
Calculates the Median Absolute Percentage Error.
$\text{MdAPE} = 100 \cdot \text{MD}_{j=1,n} \left( \frac{A_j  P_j}{A_j} \right)$where $A_j$ is the ground truth and $P_j$ is the predicted value.
More details can be found in Botchkarev 2018.
update
must receive output of the form(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.y and y_pred must be of same shape (N, ) or (N, 1) and of type float32.
Warning
Current implementation stores all input data (output and target) in as tensors before computing a metric. This can potentially lead to a memory error if the input data is larger than available RAM.
 Parameters
output_transform (Callable) – a callable that is used to transform the
Engine
’sprocess_function
’s output into the form expected by the metric. This can be useful if, for example, you have a multioutput model and you want to compute the metric with respect to one of the outputs. By default, metrics require the output as(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.
 class ignite.contrib.metrics.regression.MedianRelativeAbsoluteError(output_transform=<function MedianRelativeAbsoluteError.<lambda>>)[source]#
Calculates the Median Relative Absolute Error.
$\text{MdRAE} = \text{MD}_{j=1,n} \left( \frac{A_j  P_j}{A_j  \bar{A}} \right)$where $A_j$ is the ground truth and $P_j$ is the predicted value.
More details can be found in Botchkarev 2018.
update
must receive output of the form(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.y and y_pred must be of same shape (N, ) or (N, 1) and of type float32.
Warning
Current implementation stores all input data (output and target) in as tensors before computing a metric. This can potentially lead to a memory error if the input data is larger than available RAM.
 Parameters
output_transform (Callable) – a callable that is used to transform the
Engine
’sprocess_function
’s output into the form expected by the metric. This can be useful if, for example, you have a multioutput model and you want to compute the metric with respect to one of the outputs. By default, metrics require the output as(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.
 class ignite.contrib.metrics.regression.R2Score(output_transform=<function Metric.<lambda>>, device=device(type='cpu'))[source]#
Calculates the RSquared, the coefficient of determination.
$R^2 = 1  \frac{\sum_{j=1}^n(A_j  P_j)^2}{\sum_{j=1}^n(A_j  \bar{A})^2}$where $A_j$ is the ground truth, $P_j$ is the predicted value and $\bar{A}$ is the mean of the ground truth.
update
must receive output of the form(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.y and y_pred must be of same shape (N, ) or (N, 1) and of type float32.
Parameters are inherited from
Metric.__init__
. Parameters
output_transform (Callable) – a callable that is used to transform the
Engine
’sprocess_function
’s output into the form expected by the metric. This can be useful if, for example, you have a multioutput model and you want to compute the metric with respect to one of the outputs. By default, metrics require the output as(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.device (Union[str, device]) – specifies which device updates are accumulated on. Setting the metric’s device to be the same as your
update
arguments ensures theupdate
method is nonblocking. By default, CPU.
Changed in version 0.4.3: Works with DDP.
 class ignite.contrib.metrics.regression.WaveHedgesDistance(output_transform=<function Metric.<lambda>>, device=device(type='cpu'))[source]#
Calculates the Wave Hedges Distance.
$\text{WHD} = \sum_{j=1}^n\frac{A_j  P_j}{max(A_j, P_j)}$where, $A_j$ is the ground truth and $P_j$ is the predicted value.
More details can be found in Botchkarev 2018.
update
must receive output of the form(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.y and y_pred must be of same shape (N, ) or (N, 1).
Parameters are inherited from
Metric.__init__
. Parameters
output_transform (Callable) – a callable that is used to transform the
Engine
’sprocess_function
’s output into the form expected by the metric. This can be useful if, for example, you have a multioutput model and you want to compute the metric with respect to one of the outputs. By default, metrics require the output as(y_pred, y)
or{'y_pred': y_pred, 'y': y}
.device (Union[str, device]) – specifies which device updates are accumulated on. Setting the metric’s device to be the same as your
update
arguments ensures theupdate
method is nonblocking. By default, CPU.