Examples¶
In this section, you will find the data loading implementations (using DataPipes) of various popular datasets across different research domains. Some of the examples are implements by the PyTorch team and the implementation codes are maintained within PyTorch libraries. Others are created by members of the PyTorch community.
Audio¶
LibriSpeech¶
LibriSpeech dataset is corpus of approximately 1000 hours of 16kHz read English speech. Here is the DataPipe implementation of LibriSpeech to load the data.
Text¶
Amazon Review Polarity¶
The Amazon reviews dataset contains reviews from Amazon. Its purpose is to train text/sentiment classification models. In our DataPipe implementation of the dataset, we described every step with detailed comments to help you understand what each DataPipe is doing. We recommend having a look at this example.
IMDB¶
This is a large movie review dataset for binary sentiment classification containing 25,000 highly polar movie reviews for training and 25,00 for testing. Here is the DataPipe implementation to load the data.
SQuAD¶
SQuAD (Stanford Question Answering Dataset) is a dataset for reading comprehension. It consists of a list of questions by crowdworkers on a set of Wikipedia articles. Here are the DataPipe implementations for version 1.1 is here and version 2.0.
Additional Datasets in TorchText¶
In a separate PyTorch domain library TorchText, you will find some of the most popular datasets in the NLP field implemented as loadable datasets using DataPipes. You can find all of those NLP datasets here.
Vision¶
Caltech 101¶
The Caltech 101 dataset contains pictures of objects belonging to 101 categories. Here is the DataPipe implementation of Caltech 101.
Caltech 256¶
The Caltech 256 dataset contains 30607 images from 256 categories. Here is the DataPipe implementation of Caltech 256.
CamVid - Semantic Segmentation (community example)¶
The Cambridge-driving Labeled Video Database (CamVid) is a collection of videos with object class semantic labels, complete with metadata. The database provides ground truth labels that associate each pixel with one of 32 semantic classes. Here is a DataPipe implementation of CamVid created by our community.
laion2B-en-joined¶
The laion2B-en-joined dataset is a subset of the LAION-5B dataset containing english captions, URls pointing to images, and other metadata. It contains around 2.32 billion entries. Currently (February 2023) around 86% of the URLs still point to valid images. Here is a DataPipe implementation of laion2B-en-joined that filters out unsafe images and images with watermarks and loads the images from the URLs.
Additional Datasets in TorchVision¶
In a separate PyTorch domain library TorchVision, you will find some of the most popular datasets in the computer vision field implemented as loadable datasets using DataPipes. You can find all of those vision datasets here.
Note that these implementations are currently in the prototype phase, but they should be fully supported in the coming months. Nonetheless, they demonstrate the different ways DataPipes can be used for data loading.
Recommender System¶
Criteo 1TB Click Logs¶
The Criteo dataset contains feature values and click feedback for millions of display advertisements. It aims to benchmark algorithms for click through rate (CTR) prediction. You can find a prototype stage implementation of the dataset with DataPipes in TorchRec.
Graphs, Meshes and Point Clouds¶
TigerGraph (community example)¶
TigerGraph is a scalable graph data platform for AI and ML. You can find an implementation of graph feature engineering and machine learning with DataPipes in TorchData and data stored in a TigerGraph database, which includes computing PageRank scores in-database, pulling graph data and features with multiple DataPipes, and training a neural network using graph features in PyTorch.
MoleculeNet (community example)¶
MoleculeNet is a benchmark specially designed for testing machine learning methods of molecular properties. You can find an implementation of the HIV dataset with DataPipes in PyTorch Geometric, which includes converting SMILES strings into molecular graph representations.
Princeton ModelNet (community example)¶
The Princeton ModelNet project provides a comprehensive and clean collection of 3D CAD models across various object types. You can find an implementation of the ModelNet10 dataset with DataPipes in PyTorch Geometric, which includes reading in meshes via meshio, and sampling of points from object surfaces and dynamic graph generation via PyG’s functional transformations.
Timeseries¶
Custom DataPipe for Timeseries rolling window (community example)¶
Implementing a rolling window custom DataPipe for timeseries forecasting tasks. Here is the DataPipe implementation of a rolling window.
Using AIStore¶
Caltech 256 and Microsoft COCO (community example)¶
Listing and loading data from AIS buckets (buckets that are not 3rd party backend-based) and remote cloud buckets (3rd party backend-based cloud buckets) using AISFileLister and AISFileLoader.
Here is an example which uses AISIO DataPipe for the Caltech-256 Object Category Dataset containing 256 object categories and a total of 30607 images stored on an AIS bucket and the Microsoft COCO Dataset which has 330K images with over 200K labels of more than 1.5 million object instances across 80 object categories stored on Google Cloud.
