.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/plot_datapoints.py"
.. LINE NUMBERS ARE GIVEN BELOW.
.. only:: html
.. note::
:class: sphx-glr-download-link-note
:ref:`Go to the end <sphx_glr_download_auto_examples_plot_datapoints.py>`
to download the full example code
.. rst-class:: sphx-glr-example-title
.. _sphx_glr_auto_examples_plot_datapoints.py:
==============
Datapoints FAQ
==============
The :mod:`torchvision.datapoints` namespace was introduced together with ``torchvision.transforms.v2``. This example
showcases what these datapoints are and how they behave. This is a fairly low-level topic that most users will not need
to worry about: you do not need to understand the internals of datapoints to efficiently rely on
``torchvision.transforms.v2``. It may however be useful for advanced users trying to implement their own datasets,
transforms, or work directly with the datapoints.
.. GENERATED FROM PYTHON SOURCE LINES 12-25
.. code-block:: default
import PIL.Image
import torch
import torchvision
# We are using BETA APIs, so we deactivate the associated warning, thereby acknowledging that
# some APIs may slightly change in the future
torchvision.disable_beta_transforms_warning()
from torchvision import datapoints
.. GENERATED FROM PYTHON SOURCE LINES 26-30
What are datapoints?
--------------------
Datapoints are zero-copy tensor subclasses:
.. GENERATED FROM PYTHON SOURCE LINES 30-38
.. code-block:: default
tensor = torch.rand(3, 256, 256)
image = datapoints.Image(tensor)
assert isinstance(image, torch.Tensor)
assert image.data_ptr() == tensor.data_ptr()
.. GENERATED FROM PYTHON SOURCE LINES 39-56
Under the hood, they are needed in :mod:`torchvision.transforms.v2` to correctly dispatch to the appropriate function
for the input data.
What datapoints are supported?
------------------------------
So far :mod:`torchvision.datapoints` supports four types of datapoints:
* :class:`~torchvision.datapoints.Image`
* :class:`~torchvision.datapoints.Video`
* :class:`~torchvision.datapoints.BoundingBox`
* :class:`~torchvision.datapoints.Mask`
How do I construct a datapoint?
-------------------------------
Each datapoint class takes any tensor-like data that can be turned into a :class:`~torch.Tensor`
.. GENERATED FROM PYTHON SOURCE LINES 56-61
.. code-block:: default
image = datapoints.Image([[[[0, 1], [1, 0]]]])
print(image)
.. rst-class:: sphx-glr-script-out
.. code-block:: none
Image([[[[0, 1],
[1, 0]]]], )
.. GENERATED FROM PYTHON SOURCE LINES 62-64
Similar to other PyTorch creations ops, the constructor also takes the ``dtype``, ``device``, and ``requires_grad``
parameters.
.. GENERATED FROM PYTHON SOURCE LINES 64-69
.. code-block:: default
float_image = datapoints.Image([[[0, 1], [1, 0]]], dtype=torch.float32, requires_grad=True)
print(float_image)
.. rst-class:: sphx-glr-script-out
.. code-block:: none
Image([[[0., 1.],
[1., 0.]]], grad_fn=<AliasBackward0>, )
.. GENERATED FROM PYTHON SOURCE LINES 70-72
In addition, :class:`~torchvision.datapoints.Image` and :class:`~torchvision.datapoints.Mask` also take a
:class:`PIL.Image.Image` directly:
.. GENERATED FROM PYTHON SOURCE LINES 72-76
.. code-block:: default
image = datapoints.Image(PIL.Image.open("assets/astronaut.jpg"))
print(image.shape, image.dtype)
.. rst-class:: sphx-glr-script-out
.. code-block:: none
torch.Size([3, 512, 512]) torch.uint8
.. GENERATED FROM PYTHON SOURCE LINES 77-80
In general, the datapoints can also store additional metadata that complements the underlying tensor. For example,
:class:`~torchvision.datapoints.BoundingBox` stores the coordinate format as well as the spatial size of the
corresponding image alongside the actual values:
.. GENERATED FROM PYTHON SOURCE LINES 80-87
.. code-block:: default
bounding_box = datapoints.BoundingBox(
[17, 16, 344, 495], format=datapoints.BoundingBoxFormat.XYXY, spatial_size=image.shape[-2:]
)
print(bounding_box)
.. rst-class:: sphx-glr-script-out
.. code-block:: none
BoundingBox([ 17, 16, 344, 495], format=BoundingBoxFormat.XYXY, spatial_size=torch.Size([512, 512]))
.. GENERATED FROM PYTHON SOURCE LINES 88-103
Do I have to wrap the output of the datasets myself?
----------------------------------------------------
Only if you are using custom datasets. For the built-in ones, you can use
:func:`torchvision.datasets.wrap_dataset_for_transforms_v2`. Note that the function also supports subclasses of the
built-in datasets. Meaning, if your custom dataset subclasses from a built-in one and the output type is the same, you
also don't have to wrap manually.
How do the datapoints behave inside a computation?
--------------------------------------------------
Datapoints look and feel just like regular tensors. Everything that is supported on a plain :class:`torch.Tensor`
also works on datapoints.
Since for most operations involving datapoints, it cannot be safely inferred whether the result should retain the
datapoint type, we choose to return a plain tensor instead of a datapoint (this might change, see note below):
.. GENERATED FROM PYTHON SOURCE LINES 103-110
.. code-block:: default
assert isinstance(image, datapoints.Image)
new_image = image + 0
assert isinstance(new_image, torch.Tensor) and not isinstance(new_image, datapoints.Image)
.. GENERATED FROM PYTHON SOURCE LINES 111-123
.. note::
This "unwrapping" behaviour is something we're actively seeking feedback on. If you find this surprising or if you
have any suggestions on how to better support your use-cases, please reach out to us via this issue:
https://github.com/pytorch/vision/issues/7319
There are two exceptions to this rule:
1. The operations :meth:`~torch.Tensor.clone`, :meth:`~torch.Tensor.to`, and :meth:`~torch.Tensor.requires_grad_`
retain the datapoint type.
2. Inplace operations on datapoints cannot change the type of the datapoint they are called on. However, if you use
the flow style, the returned value will be unwrapped:
.. GENERATED FROM PYTHON SOURCE LINES 123-133
.. code-block:: default
image = datapoints.Image([[[0, 1], [1, 0]]])
new_image = image.add_(1).mul_(2)
assert isinstance(image, torch.Tensor)
print(image)
assert isinstance(new_image, torch.Tensor) and not isinstance(new_image, datapoints.Image)
assert (new_image == image).all()
.. rst-class:: sphx-glr-script-out
.. code-block:: none
Image([[[2, 4],
[4, 2]]], )
.. rst-class:: sphx-glr-timing
**Total running time of the script:** ( 0 minutes 0.046 seconds)
.. _sphx_glr_download_auto_examples_plot_datapoints.py:
.. only:: html
.. container:: sphx-glr-footer sphx-glr-footer-example
.. container:: sphx-glr-download sphx-glr-download-python
:download:`Download Python source code: plot_datapoints.py <plot_datapoints.py>`
.. container:: sphx-glr-download sphx-glr-download-jupyter
:download:`Download Jupyter notebook: plot_datapoints.ipynb <plot_datapoints.ipynb>`
.. only:: html
.. rst-class:: sphx-glr-signature
`Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_