torcheval.metrics.WindowedWeightedCalibration

class torcheval.metrics.WindowedWeightedCalibration(*, num_tasks: int = 1, max_num_updates: int = 100, enable_lifetime: bool = True, device: device | None = None)

Compute weighted calibration metric. When weight is not provided, it calculates the unweighted calibration. Its functional version is torcheval.metrics.functional.weighted_calibration().

weighted_calibration = sum(input * weight) / sum(target * weight)

Parameters:

• num_tasks (int) – Number of tasks that need WeightedCalibration calculations. Default value is 1.

• max_num_updates (int) – The max window size that can accommodate the number of updates.

• enable_lifetime (bool) – A boolean indicator whether to calculate lifetime values.

Raises:

ValueError – If value of weight is neither a float nor a int nor a torch.Tensor that matches the input tensor size.

Examples:

>>> import torch
>>> from torcheval.metrics import WindowedWeightedCalibration
>>> metric = WindowedWeightedCalibration(max_num_updates=2, enable_lifetime=False)
>>> metric.update(torch.tensor([0.8, 0.4]),torch.tensor([1, 1]))
>>> metric.update(torch.tensor([0.3, 0.8]),torch.tensor([0, 0]))
>>> metric.update(torch.tensor([0.7, 0.6]),torch.tensor([1, 0]))
>>> metric.compute()
tensor([2.4], dtype=torch.float64)

>>> metric = WindowedWeightedCalibration(max_num_updates=2, enable_lifetime=True)
>>> metric.update(torch.tensor([0.8, 0.4]),torch.tensor([1, 1]))
>>> metric.update(torch.tensor([0.3, 0.8]),torch.tensor([0, 0]))
>>> metric.update(torch.tensor([0.7, 0.6]),torch.tensor([1, 0]))
>>> metric.compute()
(
tensor([1.2], dtype=torch.float64)
tensor([2.4], dtype=torch.float64)
)

>>> metric = WindowedWeightedCalibration(num_tasks=2)
>>> metric.update(torch.tensor([[0.8, 0.4], [0.8, 0.7]]),torch.tensor([[1, 1], [0, 1]]),)
>>> metric.compute()
tensor([0.6000, 1.5000], dtype=torch.float64)

__init__(*, num_tasks: int = 1, max_num_updates: int = 100, enable_lifetime: bool = True, device: device | None = None) → None

Initialize a metric object and its internal states.

Use self._add_state() to initialize state variables of your metric class. The state variables should be either torch.Tensor, a list of torch.Tensor, a dictionary with torch.Tensor as values, or a deque of torch.Tensor.

Methods

__init__(*[, num_tasks, max_num_updates, ...])	Initialize a metric object and its internal states.
compute()	Return the weighted calibration.
load_state_dict(state_dict[, strict])	Loads metric state variables from state_dict.
merge_state(metrics)	Merge the metric state with its counterparts from other metric instances.
reset()	Reset the metric state variables to their default value.
state_dict()	Save metric state variables in state_dict.
to(device, *args, **kwargs)	Move tensors in metric state variables to device.
update(input, target[, weight])	Update the metric state with the total sum of weighted inputs and the total sum of weighted labels.

Attributes

device

The last input device of Metric.to().