torcheval.metrics.Perplexity

class torcheval.metrics.Perplexity(ignore_index: int | None = None, device: device | None = None)

Perplexity measures how well a model predicts sample data. It is calculated by:

ppl = exp (sum of negative log likelihood / number of tokens)

Its functional version is torcheval.metrics.functional.text.perplexity.

Parameters:

ignore_index (Tensor) – if specified, the target class with ‘ignore_index’ will be ignored when calculating perplexity. The default value is None.

Examples

>>> import torch
>>> from torcheval.metrics.text import Perplexity

>>> metric=Perplexity()
>>> input = torch.tensor([[[0.3659, 0.7025, 0.3104]], [[0.0097, 0.6577, 0.1947]],[[0.5659, 0.0025, 0.0104]], [[0.9097, 0.0577, 0.7947]]])
>>> target = torch.tensor([[2],  [1], [2],  [1]])
>>> metric.update(input, target)
>>> metric.compute()
tensor(3.5257, dtype=torch.float64)

>>> metric=Perplexity(ignore_index=1)
>>> input = torch.tensor([[[0.3659, 0.7025, 0.3104]], [[0.0097, 0.6577, 0.1947]],[[0.5659, 0.0025, 0.0104]], [[0.9097, 0.0577, 0.7947]]])
>>> target = torch.tensor([[2],  [1], [2],  [1]])
>>> metric.update(input, target)
>>> metric.compute()
tensor(3.6347, dtype=torch.float64)

>>> metric1=Perplexity()
>>> input = torch.tensor([[[0.5659, 0.0025, 0.0104]], [[0.9097, 0.0577, 0.7947]]])
>>> target = torch.tensor([[2],  [1], ])
>>> metric1.update(input, target)
>>> metric1.compute()
tensor(4.5051, dtype=torch.float64)

>>> metric2=Perplexity()
>>> input = torch.tensor([[[0.3659, 0.7025, 0.3104]], [[0.0097, 0.6577, 0.1947]]])
>>> target = torch.tensor([[2],  [1]])
>>> metric2.update(input, target)
>>> metric2.compute())
tensor(2.7593, dtype=torch.float64)

>>> metric1.merge_state([metric2])
>>> metric1.compute())
tensor(3.5257, dtype=torch.float64)

__init__(ignore_index: int | None = None, device: device | None = None) → None

Initialize a metric object and its internal states.

Use self._add_state() to initialize state variables of your metric class. The state variables should be either torch.Tensor, a list of torch.Tensor, a dictionary with torch.Tensor as values, or a deque of torch.Tensor.

Methods

__init__([ignore_index, device])	Initialize a metric object and its internal states.
compute()	Calculates perplexity based on sum_log_probs and num_total.
load_state_dict(state_dict[, strict])	Loads metric state variables from state_dict.
merge_state(metrics)	Merge the metric state with its counterparts from other metric instances.
reset()	Reset the metric state variables to their default value.
state_dict()	Save metric state variables in state_dict.
to(device, *args, **kwargs)	Move tensors in metric state variables to device.
update(input, target)	Update the metric state with new inputs.

Attributes

device

The last input device of Metric.to().