torcheval.metrics.WordInformationLost

class torcheval.metrics.WordInformationLost(device: Optional[device] = None)

Word Information Lost (WIL) is a metric of the performance of an automatic speech recognition system. This value indicates the percentage of words that were incorrectly predicted between a set of ground-truth sentences and a set of hypothesis sentences. The lower the value, the better the performance of the ASR system with a WordInformationLost of 0 being a perfect score. Word Information Lost rate can then be computed as:

\[wil = 1 - \frac{C}{N} * \frac{C}{P}\]

where:

• \(C\) is the number of correct words,
• \(N\) is the number of words in the reference
• \(P\) is the number of words in the prediction

Its functional version is torcheval.metrics.functional.word_information_lost().

Examples

>>> from torcheval.metrics.text import WordInformationLost
>>> preds = ["this is the prediction", "there is an other sample"]
>>> target = ["this is the reference", "there is another one"]
>>> metric = WordInformationLost()
>>> metric(preds, target)
tensor(0.6528)

__init__(device: Optional[device] = None) → None

Initialize a metric object and its internal states.

Use self._add_state() to initialize state variables of your metric class. The state variables should be either torch.Tensor, a list of torch.Tensor, or a dictionary with torch.Tensor as values

Methods

__init__([device])	Initialize a metric object and its internal states.
compute()	Calculate the Word Information Lost.
load_state_dict(state_dict[, strict])	Loads metric state variables from state_dict.
merge_state(metrics)	Merge the metric state with its counterparts from other metric instances.
reset()	Reset the metric state variables to their default value.
state_dict()	Save metric state variables in state_dict.
to(device, *args, **kwargs)	Move tensors in metric state variables to device.
update(input, target)	Store predictions/references for computing Word Information Lost scores.

Attributes

device

The last input device of Metric.to().