A library with simple and straightforward tooling for model evaluations and a delightful user experience. At a high level TorchEval:
Contains a rich collection of high performance metric calculations out of the box. We utilize vectorization and GPU acceleration where possible via PyTorch.
Integrates seamlessly with distributed training and tools using torch.distributed
Is designed with extensibility in mind: you have the freedom to easily create your own metrics and leverage our toolkit.
Provides tools for profiling memory and compute requirements for PyTorch based models.
TorchEval can be installed from PyPi via
pip install torcheval
or from github
git clone https://github.com/pytorch/torcheval cd torcheval pip install -r requirements.txt python setup.py install
TorchEval provides two interfaces to each metric. If you are working in a single process environment, it is simplest to use metrics from the
functional submodule. These can be found in
from torcheval.metrics.functional import binary_f1_score predictions = model(inputs) f1_score = binary_f1_score(predictions, targets)
We can use the same metric in the class based route, which provides tools that make computation simple in a multi-process setting. On a single device, you can use the class based metrics as follows:
from torcheval.metrics import BinaryF1Score predictions = model(inputs) metric = BinaryF1Score() metric.update(predictions, targets) f1_score = metric.compute()
In a multi-process setting, the data from each process must be synchronized to compute the metric across the full dataset. To do this, simply replace
from torcheval.metrics import BinaryF1Score from torcheval.metrics.toolkit import sync_and_compute predictions = model(inputs) metric = BinaryF1Score() metric.update(predictions, targets) f1_score = sync_and_compute(metric)
Read more about the class based method in the distributed example.
Check out the guides explaining the compute example
Check out the distributed example
Check out how to make your own metric
Indices and tables¶
- Functional Metrics
- Metric Toolkit
- Module Tools