.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "beginner/basics/data_tutorial.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        Click :ref:`here <sphx_glr_download_beginner_basics_data_tutorial.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_beginner_basics_data_tutorial.py:


`Learn the Basics <intro.html>`_ ||
`Quickstart <quickstart_tutorial.html>`_ ||
`Tensors <tensorqs_tutorial.html>`_ ||
**Datasets & DataLoaders** ||
`Transforms <transforms_tutorial.html>`_ ||
`Build Model <buildmodel_tutorial.html>`_ ||
`Autograd <autogradqs_tutorial.html>`_ ||
`Optimization <optimization_tutorial.html>`_ ||
`Save & Load Model <saveloadrun_tutorial.html>`_

Datasets & DataLoaders
======================

.. GENERATED FROM PYTHON SOURCE LINES 18-32

Code for processing data samples can get messy and hard to maintain; we ideally want our dataset code
to be decoupled from our model training code for better readability and modularity.
PyTorch provides two data primitives: ``torch.utils.data.DataLoader`` and ``torch.utils.data.Dataset``
that allow you to use pre-loaded datasets as well as your own data.
``Dataset`` stores the samples and their corresponding labels, and ``DataLoader`` wraps an iterable around
the ``Dataset`` to enable easy access to the samples.

PyTorch domain libraries provide a number of pre-loaded datasets (such as FashionMNIST) that
subclass ``torch.utils.data.Dataset`` and implement functions specific to the particular data.
They can be used to prototype and benchmark your model. You can find them
here: `Image Datasets <https://pytorch.org/vision/stable/datasets.html>`_,
`Text Datasets  <https://pytorch.org/text/stable/datasets.html>`_, and
`Audio Datasets <https://pytorch.org/audio/stable/datasets.html>`_


.. GENERATED FROM PYTHON SOURCE LINES 34-46

Loading a Dataset
-------------------

Here is an example of how to load the `Fashion-MNIST <https://research.zalando.com/project/fashion_mnist/fashion_mnist/>`_ dataset from TorchVision.
Fashion-MNIST is a dataset of Zalando’s article images consisting of 60,000 training examples and 10,000 test examples.
Each example comprises a 28×28 grayscale image and an associated label from one of 10 classes.

We load the `FashionMNIST Dataset <https://pytorch.org/vision/stable/datasets.html#fashion-mnist>`_ with the following parameters:
 - ``root`` is the path where the train/test data is stored,
 - ``train`` specifies training or test dataset,
 - ``download=True`` downloads the data from the internet if it's not available at ``root``.
 - ``transform`` and ``target_transform`` specify the feature and label transformations

.. GENERATED FROM PYTHON SOURCE LINES 46-70

.. code-block:: default



    import torch
    from torch.utils.data import Dataset
    from torchvision import datasets
    from torchvision.transforms import ToTensor
    import matplotlib.pyplot as plt


    training_data = datasets.FashionMNIST(
        root="data",
        train=True,
        download=True,
        transform=ToTensor()
    )

    test_data = datasets.FashionMNIST(
        root="data",
        train=False,
        download=True,
        transform=ToTensor()
    )






.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz
    Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz to data/FashionMNIST/raw/train-images-idx3-ubyte.gz

      0%|          | 0/26421880 [00:00<?, ?it/s]
      0%|          | 65536/26421880 [00:00<01:12, 363434.07it/s]
      1%|          | 229376/26421880 [00:00<00:38, 683154.28it/s]
      3%|3         | 884736/26421880 [00:00<00:10, 2528991.62it/s]
      7%|6         | 1835008/26421880 [00:00<00:06, 3867061.97it/s]
     22%|##1       | 5701632/26421880 [00:00<00:01, 13093642.12it/s]
     37%|###7      | 9830400/26421880 [00:00<00:00, 17476123.85it/s]
     54%|#####4    | 14286848/26421880 [00:01<00:00, 24123407.60it/s]
     70%|######9   | 18415616/26421880 [00:01<00:00, 24112352.48it/s]
     87%|########6 | 22872064/26421880 [00:01<00:00, 28999642.65it/s]
    100%|##########| 26421880/26421880 [00:01<00:00, 19275562.62it/s]
    Extracting data/FashionMNIST/raw/train-images-idx3-ubyte.gz to data/FashionMNIST/raw

    Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz
    Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz to data/FashionMNIST/raw/train-labels-idx1-ubyte.gz

      0%|          | 0/29515 [00:00<?, ?it/s]
    100%|##########| 29515/29515 [00:00<00:00, 325727.15it/s]
    Extracting data/FashionMNIST/raw/train-labels-idx1-ubyte.gz to data/FashionMNIST/raw

    Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz
    Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz to data/FashionMNIST/raw/t10k-images-idx3-ubyte.gz

      0%|          | 0/4422102 [00:00<?, ?it/s]
      1%|1         | 65536/4422102 [00:00<00:11, 363060.61it/s]
      5%|5         | 229376/4422102 [00:00<00:06, 682828.35it/s]
     21%|##1       | 950272/4422102 [00:00<00:01, 2190535.66it/s]
     87%|########6 | 3833856/4422102 [00:00<00:00, 7617757.26it/s]
    100%|##########| 4422102/4422102 [00:00<00:00, 6092265.42it/s]
    Extracting data/FashionMNIST/raw/t10k-images-idx3-ubyte.gz to data/FashionMNIST/raw

    Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz
    Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz to data/FashionMNIST/raw/t10k-labels-idx1-ubyte.gz

      0%|          | 0/5148 [00:00<?, ?it/s]
    100%|##########| 5148/5148 [00:00<00:00, 39911787.42it/s]
    Extracting data/FashionMNIST/raw/t10k-labels-idx1-ubyte.gz to data/FashionMNIST/raw





.. GENERATED FROM PYTHON SOURCE LINES 71-76

Iterating and Visualizing the Dataset
-------------------------------------

We can index ``Datasets`` manually like a list: ``training_data[index]``.
We use ``matplotlib`` to visualize some samples in our training data.

.. GENERATED FROM PYTHON SOURCE LINES 76-100

.. code-block:: default


    labels_map = {
        0: "T-Shirt",
        1: "Trouser",
        2: "Pullover",
        3: "Dress",
        4: "Coat",
        5: "Sandal",
        6: "Shirt",
        7: "Sneaker",
        8: "Bag",
        9: "Ankle Boot",
    }
    figure = plt.figure(figsize=(8, 8))
    cols, rows = 3, 3
    for i in range(1, cols * rows + 1):
        sample_idx = torch.randint(len(training_data), size=(1,)).item()
        img, label = training_data[sample_idx]
        figure.add_subplot(rows, cols, i)
        plt.title(labels_map[label])
        plt.axis("off")
        plt.imshow(img.squeeze(), cmap="gray")
    plt.show()




.. image-sg:: /beginner/basics/images/sphx_glr_data_tutorial_001.png
   :alt: Ankle Boot, Shirt, Bag, Ankle Boot, Trouser, Sandal, Coat, Sandal, Pullover
   :srcset: /beginner/basics/images/sphx_glr_data_tutorial_001.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 101-104

..
 .. figure:: /_static/img/basics/fashion_mnist.png
   :alt: fashion_mnist

.. GENERATED FROM PYTHON SOURCE LINES 107-109

--------------


.. GENERATED FROM PYTHON SOURCE LINES 111-119

Creating a Custom Dataset for your files
---------------------------------------------------

A custom Dataset class must implement three functions: `__init__`, `__len__`, and `__getitem__`.
Take a look at this implementation; the FashionMNIST images are stored
in a directory ``img_dir``, and their labels are stored separately in a CSV file ``annotations_file``.

In the next sections, we'll break down what's happening in each of these functions.

.. GENERATED FROM PYTHON SOURCE LINES 119-146

.. code-block:: default



    import os
    import pandas as pd
    from torchvision.io import read_image

    class CustomImageDataset(Dataset):
        def __init__(self, annotations_file, img_dir, transform=None, target_transform=None):
            self.img_labels = pd.read_csv(annotations_file)
            self.img_dir = img_dir
            self.transform = transform
            self.target_transform = target_transform

        def __len__(self):
            return len(self.img_labels)

        def __getitem__(self, idx):
            img_path = os.path.join(self.img_dir, self.img_labels.iloc[idx, 0])
            image = read_image(img_path)
            label = self.img_labels.iloc[idx, 1]
            if self.transform:
                image = self.transform(image)
            if self.target_transform:
                label = self.target_transform(label)
            return image, label









.. GENERATED FROM PYTHON SOURCE LINES 147-160

``__init__``
^^^^^^^^^^^^^^^^^^^^

The __init__ function is run once when instantiating the Dataset object. We initialize
the directory containing the images, the annotations file, and both transforms (covered
in more detail in the next section).

The labels.csv file looks like: ::

    tshirt1.jpg, 0
    tshirt2.jpg, 0
    ......
    ankleboot999.jpg, 9

.. GENERATED FROM PYTHON SOURCE LINES 160-169

.. code-block:: default



    def __init__(self, annotations_file, img_dir, transform=None, target_transform=None):
        self.img_labels = pd.read_csv(annotations_file)
        self.img_dir = img_dir
        self.transform = transform
        self.target_transform = target_transform









.. GENERATED FROM PYTHON SOURCE LINES 170-176

``__len__``
^^^^^^^^^^^^^^^^^^^^

The __len__ function returns the number of samples in our dataset.

Example:

.. GENERATED FROM PYTHON SOURCE LINES 176-182

.. code-block:: default



    def __len__(self):
        return len(self.img_labels)









.. GENERATED FROM PYTHON SOURCE LINES 183-190

``__getitem__``
^^^^^^^^^^^^^^^^^^^^

The __getitem__ function loads and returns a sample from the dataset at the given index ``idx``.
Based on the index, it identifies the image's location on disk, converts that to a tensor using ``read_image``, retrieves the
corresponding label from the csv data in ``self.img_labels``, calls the transform functions on them (if applicable), and returns the
tensor image and corresponding label in a tuple.

.. GENERATED FROM PYTHON SOURCE LINES 190-202

.. code-block:: default


    def __getitem__(self, idx):
        img_path = os.path.join(self.img_dir, self.img_labels.iloc[idx, 0])
        image = read_image(img_path)
        label = self.img_labels.iloc[idx, 1]
        if self.transform:
            image = self.transform(image)
        if self.target_transform:
            label = self.target_transform(label)
        return image, label









.. GENERATED FROM PYTHON SOURCE LINES 203-205

--------------


.. GENERATED FROM PYTHON SOURCE LINES 208-215

Preparing your data for training with DataLoaders
-------------------------------------------------
The ``Dataset`` retrieves our dataset's features and labels one sample at a time. While training a model, we typically want to
pass samples in "minibatches", reshuffle the data at every epoch to reduce model overfitting, and use Python's ``multiprocessing`` to
speed up data retrieval.

``DataLoader`` is an iterable that abstracts this complexity for us in an easy API.

.. GENERATED FROM PYTHON SOURCE LINES 215-221

.. code-block:: default


    from torch.utils.data import DataLoader

    train_dataloader = DataLoader(training_data, batch_size=64, shuffle=True)
    test_dataloader = DataLoader(test_data, batch_size=64, shuffle=True)








.. GENERATED FROM PYTHON SOURCE LINES 222-229

Iterate through the DataLoader
-------------------------------

We have loaded that dataset into the ``DataLoader`` and can iterate through the dataset as needed.
Each iteration below returns a batch of ``train_features`` and ``train_labels`` (containing ``batch_size=64`` features and labels respectively).
Because we specified ``shuffle=True``, after we iterate over all batches the data is shuffled (for finer-grained control over
the data loading order, take a look at `Samplers <https://pytorch.org/docs/stable/data.html#data-loading-order-and-sampler>`_).

.. GENERATED FROM PYTHON SOURCE LINES 229-240

.. code-block:: default


    # Display image and label.
    train_features, train_labels = next(iter(train_dataloader))
    print(f"Feature batch shape: {train_features.size()}")
    print(f"Labels batch shape: {train_labels.size()}")
    img = train_features[0].squeeze()
    label = train_labels[0]
    plt.imshow(img, cmap="gray")
    plt.show()
    print(f"Label: {label}")




.. image-sg:: /beginner/basics/images/sphx_glr_data_tutorial_002.png
   :alt: data tutorial
   :srcset: /beginner/basics/images/sphx_glr_data_tutorial_002.png
   :class: sphx-glr-single-img


.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Feature batch shape: torch.Size([64, 1, 28, 28])
    Labels batch shape: torch.Size([64])
    Label: 5




.. GENERATED FROM PYTHON SOURCE LINES 241-243

--------------


.. GENERATED FROM PYTHON SOURCE LINES 245-248

Further Reading
----------------
- `torch.utils.data API <https://pytorch.org/docs/stable/data.html>`_


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 0 minutes  4.855 seconds)


.. _sphx_glr_download_beginner_basics_data_tutorial.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example


    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: data_tutorial.py <data_tutorial.py>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: data_tutorial.ipynb <data_tutorial.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_