.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "tutorials/nvdec_tutorial.py"
.. LINE NUMBERS ARE GIVEN BELOW.
.. only:: html
.. note::
:class: sphx-glr-download-link-note
Click :ref:`here <sphx_glr_download_tutorials_nvdec_tutorial.py>`
to download the full example code
.. rst-class:: sphx-glr-example-title
.. _sphx_glr_tutorials_nvdec_tutorial.py:
Accelerated video decoding with NVDEC
=====================================
.. _nvdec_tutorial:
**Author**: `Moto Hira <moto@meta.com>`__
This tutorial shows how to use NVIDIA’s hardware video decoder (NVDEC)
with TorchAudio, and how it improves the performance of video decoding.
.. GENERATED FROM PYTHON SOURCE LINES 14-23
.. note::
This tutorial requires FFmpeg libraries compiled with HW
acceleration enabled.
Please refer to
:ref:`Enabling GPU video decoder/encoder <enabling_hw_decoder>`
for how to build FFmpeg with HW acceleration.
.. GENERATED FROM PYTHON SOURCE LINES 24-31
.. code-block:: default
import torch
import torchaudio
print(torch.__version__)
print(torchaudio.__version__)
.. rst-class:: sphx-glr-script-out
.. code-block:: none
2.6.0
2.6.0
.. GENERATED FROM PYTHON SOURCE LINES 33-39
.. code-block:: default
import os
import time
import matplotlib.pyplot as plt
from torchaudio.io import StreamReader
.. GENERATED FROM PYTHON SOURCE LINES 40-46
Check the prerequisites
-----------------------
First, we check that TorchAudio correctly detects FFmpeg libraries
that support HW decoder/encoder.
.. GENERATED FROM PYTHON SOURCE LINES 47-50
.. code-block:: default
from torchaudio.utils import ffmpeg_utils
.. GENERATED FROM PYTHON SOURCE LINES 52-57
.. code-block:: default
print("FFmpeg Library versions:")
for k, ver in ffmpeg_utils.get_versions().items():
print(f" {k}:\t{'.'.join(str(v) for v in ver)}")
.. rst-class:: sphx-glr-script-out
.. code-block:: none
FFmpeg Library versions:
libavcodec: 60.3.100
libavdevice: 60.1.100
libavfilter: 9.3.100
libavformat: 60.3.100
libavutil: 58.2.100
.. GENERATED FROM PYTHON SOURCE LINES 59-65
.. code-block:: default
print("Available NVDEC Decoders:")
for k in ffmpeg_utils.get_video_decoders().keys():
if "cuvid" in k:
print(f" - {k}")
.. rst-class:: sphx-glr-script-out
.. code-block:: none
Available NVDEC Decoders:
- av1_cuvid
- h264_cuvid
- hevc_cuvid
- mjpeg_cuvid
- mpeg1_cuvid
- mpeg2_cuvid
- mpeg4_cuvid
- vc1_cuvid
- vp8_cuvid
- vp9_cuvid
.. GENERATED FROM PYTHON SOURCE LINES 67-71
.. code-block:: default
print("Avaialbe GPU:")
print(torch.cuda.get_device_properties(0))
.. rst-class:: sphx-glr-script-out
.. code-block:: none
Avaialbe GPU:
_CudaDeviceProperties(name='NVIDIA A10G', major=8, minor=6, total_memory=22502MB, multi_processor_count=80, uuid=3a6a8555-efc9-d0dc-972b-36624af6fad8, L2_cache_size=6MB)
.. GENERATED FROM PYTHON SOURCE LINES 72-84
We will use the following video which has the following properties;
- Codec: H.264
- Resolution: 960x540
- FPS: 29.97
- Pixel format: YUV420P
.. raw:: html
<video style="max-width: 100%" controls>
<source src="https://download.pytorch.org/torchaudio/tutorial-assets/stream-api/NASAs_Most_Scientifically_Complex_Space_Observatory_Requires_Precision-MP4_small.mp4" type="video/mp4">
</video>
.. GENERATED FROM PYTHON SOURCE LINES 88-93
.. code-block:: default
src = torchaudio.utils.download_asset(
"tutorial-assets/stream-api/NASAs_Most_Scientifically_Complex_Space_Observatory_Requires_Precision-MP4_small.mp4"
)
.. rst-class:: sphx-glr-script-out
.. code-block:: none
0%| | 0.00/31.8M [00:00<?, ?B/s]
100%|##########| 31.8M/31.8M [00:00<00:00, 545MB/s]
.. GENERATED FROM PYTHON SOURCE LINES 94-101
Decoding videos with NVDEC
--------------------------
To use HW video decoder, you need to specify the HW decoder when
defining the output video stream by passing ``decoder`` option to
:py:meth:`~torchaudio.io.StreamReader.add_video_stream` method.
.. GENERATED FROM PYTHON SOURCE LINES 101-107
.. code-block:: default
s = StreamReader(src)
s.add_video_stream(5, decoder="h264_cuvid")
s.fill_buffer()
(video,) = s.pop_chunks()
.. GENERATED FROM PYTHON SOURCE LINES 108-109
The video frames are decoded and returned as tensor of NCHW format.
.. GENERATED FROM PYTHON SOURCE LINES 110-113
.. code-block:: default
print(video.shape, video.dtype)
.. rst-class:: sphx-glr-script-out
.. code-block:: none
torch.Size([5, 3, 540, 960]) torch.uint8
.. GENERATED FROM PYTHON SOURCE LINES 114-116
By default, the decoded frames are sent back to CPU memory, and
CPU tensors are created.
.. GENERATED FROM PYTHON SOURCE LINES 117-120
.. code-block:: default
print(video.device)
.. rst-class:: sphx-glr-script-out
.. code-block:: none
cpu
.. GENERATED FROM PYTHON SOURCE LINES 121-134
By specifying ``hw_accel`` option, you can convert the decoded frames
to CUDA tensor.
``hw_accel`` option takes string values and pass it
to :py:class:`torch.device`.
.. note::
Currently, ``hw_accel`` option and
:py:meth:`~torchaudio.io.StreamReader.add_basic_video_stream`
are not compatible. ``add_basic_video_stream`` adds post-decoding
process, which is designed for frames in CPU memory.
Please use :py:meth:`~torchaudio.io.StreamReader.add_video_stream`.
.. GENERATED FROM PYTHON SOURCE LINES 135-144
.. code-block:: default
s = StreamReader(src)
s.add_video_stream(5, decoder="h264_cuvid", hw_accel="cuda:0")
s.fill_buffer()
(video,) = s.pop_chunks()
print(video.shape, video.dtype, video.device)
.. rst-class:: sphx-glr-script-out
.. code-block:: none
torch.Size([5, 3, 540, 960]) torch.uint8 cuda:0
.. GENERATED FROM PYTHON SOURCE LINES 145-180
.. note::
When there are multiple of GPUs available, ``StreamReader`` by
default uses the first GPU. You can change this by providing
``"gpu"`` option.
.. code::
# Video data is sent to CUDA device 0, decoded and
# converted on the same device.
s.add_video_stream(
...,
decoder="h264_cuvid",
decoder_option={"gpu": "0"},
hw_accel="cuda:0",
)
.. note::
``"gpu"`` option and ``hw_accel`` option can be specified
independently. If they do not match, decoded frames are
transfered to the device specified by ``hw_accell``
automatically.
.. code::
# Video data is sent to CUDA device 0, and decoded there.
# Then it is transfered to CUDA device 1, and converted to
# CUDA tensor.
s.add_video_stream(
...,
decoder="h264_cuvid",
decoder_option={"gpu": "0"},
hw_accel="cuda:1",
)
.. GENERATED FROM PYTHON SOURCE LINES 182-190
Visualization
-------------
Let's look at the frames decoded by HW decoder and compare them
against equivalent results from software decoders.
The following function seeks into the given timestamp and decode one
frame with the specificed decoder.
.. GENERATED FROM PYTHON SOURCE LINES 190-201
.. code-block:: default
def test_decode(decoder: str, seek: float):
s = StreamReader(src)
s.seek(seek)
s.add_video_stream(1, decoder=decoder)
s.fill_buffer()
(video,) = s.pop_chunks()
return video[0]
.. GENERATED FROM PYTHON SOURCE LINES 203-209
.. code-block:: default
timestamps = [12, 19, 45, 131, 180]
cpu_frames = [test_decode(decoder="h264", seek=ts) for ts in timestamps]
cuda_frames = [test_decode(decoder="h264_cuvid", seek=ts) for ts in timestamps]
.. GENERATED FROM PYTHON SOURCE LINES 210-216
.. note::
Currently, HW decoder does not support colorspace conversion.
Decoded frames are YUV format.
The following function performs YUV to RGB covnersion
(and axis shuffling for plotting).
.. GENERATED FROM PYTHON SOURCE LINES 217-238
.. code-block:: default
def yuv_to_rgb(frames):
frames = frames.cpu().to(torch.float)
y = frames[..., 0, :, :]
u = frames[..., 1, :, :]
v = frames[..., 2, :, :]
y /= 255
u = u / 255 - 0.5
v = v / 255 - 0.5
r = y + 1.14 * v
g = y + -0.396 * u - 0.581 * v
b = y + 2.029 * u
rgb = torch.stack([r, g, b], -1)
rgb = (rgb * 255).clamp(0, 255).to(torch.uint8)
return rgb.numpy()
.. GENERATED FROM PYTHON SOURCE LINES 239-241
Now we visualize the resutls.
.. GENERATED FROM PYTHON SOURCE LINES 242-259
.. code-block:: default
def plot():
n_rows = len(timestamps)
fig, axes = plt.subplots(n_rows, 2, figsize=[12.8, 16.0])
for i in range(n_rows):
axes[i][0].imshow(yuv_to_rgb(cpu_frames[i]))
axes[i][1].imshow(yuv_to_rgb(cuda_frames[i]))
axes[0][0].set_title("Software decoder")
axes[0][1].set_title("HW decoder")
plt.setp(axes, xticks=[], yticks=[])
plt.tight_layout()
plot()
.. image-sg:: /tutorials/images/sphx_glr_nvdec_tutorial_001.png
:alt: Software decoder, HW decoder
:srcset: /tutorials/images/sphx_glr_nvdec_tutorial_001.png
:class: sphx-glr-single-img
.. GENERATED FROM PYTHON SOURCE LINES 260-263
They are indistinguishable to the eyes of the author.
Feel free to let us know if you spot something. :)
.. GENERATED FROM PYTHON SOURCE LINES 267-285
HW resizing and cropping
------------------------
You can use ``decoder_option`` argument to provide decoder-specific
options.
The following options are often relevant in preprocessing.
- ``resize``: Resize the frame into ``(width)x(height)``.
- ``crop``: Crop the frame ``(top)x(bottom)x(left)x(right)``.
Note that the specified values are the amount of rows/columns removed.
The final image size is ``(width - left - right)x(height - top -bottom)``.
If ``crop`` and ``resize`` options are used together,
``crop`` is performed first.
For other available options, please run
``ffmpeg -h decoder=h264_cuvid``.
.. GENERATED FROM PYTHON SOURCE LINES 285-297
.. code-block:: default
def test_options(option):
s = StreamReader(src)
s.seek(87)
s.add_video_stream(1, decoder="h264_cuvid", hw_accel="cuda:0", decoder_option=option)
s.fill_buffer()
(video,) = s.pop_chunks()
print(f"Option: {option}:\t{video.shape}")
return video[0]
.. GENERATED FROM PYTHON SOURCE LINES 299-306
.. code-block:: default
original = test_options(option=None)
resized = test_options(option={"resize": "480x270"})
cropped = test_options(option={"crop": "135x135x240x240"})
cropped_and_resized = test_options(option={"crop": "135x135x240x240", "resize": "640x360"})
.. rst-class:: sphx-glr-script-out
.. code-block:: none
Option: None: torch.Size([1, 3, 540, 960])
Option: {'resize': '480x270'}: torch.Size([1, 3, 270, 480])
Option: {'crop': '135x135x240x240'}: torch.Size([1, 3, 270, 480])
Option: {'crop': '135x135x240x240', 'resize': '640x360'}: torch.Size([1, 3, 360, 640])
.. GENERATED FROM PYTHON SOURCE LINES 308-327
.. code-block:: default
def plot():
fig, axes = plt.subplots(2, 2, figsize=[12.8, 9.6])
axes[0][0].imshow(yuv_to_rgb(original))
axes[0][1].imshow(yuv_to_rgb(resized))
axes[1][0].imshow(yuv_to_rgb(cropped))
axes[1][1].imshow(yuv_to_rgb(cropped_and_resized))
axes[0][0].set_title("Original")
axes[0][1].set_title("Resized")
axes[1][0].set_title("Cropped")
axes[1][1].set_title("Cropped and resized")
plt.tight_layout()
return fig
plot()
.. image-sg:: /tutorials/images/sphx_glr_nvdec_tutorial_002.png
:alt: Original, Resized, Cropped, Cropped and resized
:srcset: /tutorials/images/sphx_glr_nvdec_tutorial_002.png
:class: sphx-glr-single-img
.. rst-class:: sphx-glr-script-out
.. code-block:: none
<Figure size 1280x960 with 4 Axes>
.. GENERATED FROM PYTHON SOURCE LINES 328-350
Comparing resizing methods
--------------------------
Unlike software scaling, NVDEC does not provide an option to choose
the scaling algorithm.
In ML applicatoins, it is often necessary to construct a
preprocessing pipeline with a similar numerical property.
So here we compare the result of hardware resizing with software
resizing of different algorithms.
We will use the following video, which contains the test pattern
generated using the following command.
.. code::
ffmpeg -y -f lavfi -t 12.05 -i mptestsrc -movflags +faststart mptestsrc.mp4
.. raw:: html
<video style="max-width: 100%" controls>
<source src="https://download.pytorch.org/torchaudio/tutorial-assets/mptestsrc.mp4" type="video/mp4">
</video>
.. GENERATED FROM PYTHON SOURCE LINES 354-358
.. code-block:: default
test_src = torchaudio.utils.download_asset("tutorial-assets/mptestsrc.mp4")
.. rst-class:: sphx-glr-script-out
.. code-block:: none
0%| | 0.00/232k [00:00<?, ?B/s]
100%|##########| 232k/232k [00:00<00:00, 41.6MB/s]
.. GENERATED FROM PYTHON SOURCE LINES 359-362
The following function decodes video and
apply the specified scaling algorithm.
.. GENERATED FROM PYTHON SOURCE LINES 362-374
.. code-block:: default
def decode_resize_ffmpeg(mode, height, width, seek):
filter_desc = None if mode is None else f"scale={width}:{height}:sws_flags={mode}"
s = StreamReader(test_src)
s.add_video_stream(1, filter_desc=filter_desc)
s.seek(seek)
s.fill_buffer()
(chunk,) = s.pop_chunks()
return chunk
.. GENERATED FROM PYTHON SOURCE LINES 375-377
The following function uses HW decoder to decode video and resize.
.. GENERATED FROM PYTHON SOURCE LINES 377-388
.. code-block:: default
def decode_resize_cuvid(height, width, seek):
s = StreamReader(test_src)
s.add_video_stream(1, decoder="h264_cuvid", decoder_option={"resize": f"{width}x{height}"}, hw_accel="cuda:0")
s.seek(seek)
s.fill_buffer()
(chunk,) = s.pop_chunks()
return chunk.cpu()
.. GENERATED FROM PYTHON SOURCE LINES 389-390
Now we execute them and visualize the resulting frames.
.. GENERATED FROM PYTHON SOURCE LINES 390-406
.. code-block:: default
params = {"height": 224, "width": 224, "seek": 3}
frames = [
decode_resize_ffmpeg(None, **params),
decode_resize_ffmpeg("neighbor", **params),
decode_resize_ffmpeg("bilinear", **params),
decode_resize_ffmpeg("bicubic", **params),
decode_resize_cuvid(**params),
decode_resize_ffmpeg("spline", **params),
decode_resize_ffmpeg("lanczos:param0=1", **params),
decode_resize_ffmpeg("lanczos:param0=3", **params),
decode_resize_ffmpeg("lanczos:param0=5", **params),
]
.. GENERATED FROM PYTHON SOURCE LINES 408-432
.. code-block:: default
def plot():
fig, axes = plt.subplots(3, 3, figsize=[12.8, 15.2])
for i, f in enumerate(frames):
h, w = f.shape[2:4]
f = f[..., : h // 4, : w // 4]
axes[i // 3][i % 3].imshow(yuv_to_rgb(f[0]))
axes[0][0].set_title("Original")
axes[0][1].set_title("nearest neighbor")
axes[0][2].set_title("bilinear")
axes[1][0].set_title("bicubic")
axes[1][1].set_title("NVDEC")
axes[1][2].set_title("spline")
axes[2][0].set_title("lanczos(1)")
axes[2][1].set_title("lanczos(3)")
axes[2][2].set_title("lanczos(5)")
plt.setp(axes, xticks=[], yticks=[])
plt.tight_layout()
plot()
.. image-sg:: /tutorials/images/sphx_glr_nvdec_tutorial_003.png
:alt: Original, nearest neighbor, bilinear, bicubic, NVDEC, spline, lanczos(1), lanczos(3), lanczos(5)
:srcset: /tutorials/images/sphx_glr_nvdec_tutorial_003.png
:class: sphx-glr-single-img
.. GENERATED FROM PYTHON SOURCE LINES 433-436
None of them is exactly the same. To the eyes of authors, lanczos(1)
appears to be most similar to NVDEC.
The bicubic looks close as well.
.. GENERATED FROM PYTHON SOURCE LINES 438-444
Benchmark NVDEC with StreamReader
---------------------------------
In this section, we compare the performace of software video
decoding and HW video decoding.
.. GENERATED FROM PYTHON SOURCE LINES 447-466
Decode as CUDA frames
---------------------
First, we compare the time it takes for software decoder and
hardware encoder to decode the same video.
To make the result comparable, when using software decoder, we move
the resulting tensor to CUDA.
The procedures to test look like the following
- Use hardware decoder and place data on CUDA directly
- Use software decoder, generate CPU Tensors and move them to CUDA.
.. note:
Because HW decoder currently only supports reading videos as
YUV444P format, we decode frames into YUV444P format for the case of
software decoder as well.
.. GENERATED FROM PYTHON SOURCE LINES 469-470
The following function implements the hardware decoder test case.
.. GENERATED FROM PYTHON SOURCE LINES 470-488
.. code-block:: default
def test_decode_cuda(src, decoder, hw_accel="cuda", frames_per_chunk=5):
s = StreamReader(src)
s.add_video_stream(frames_per_chunk, decoder=decoder, hw_accel=hw_accel)
num_frames = 0
chunk = None
t0 = time.monotonic()
for (chunk,) in s.stream():
num_frames += chunk.shape[0]
elapsed = time.monotonic() - t0
print(f" - Shape: {chunk.shape}")
fps = num_frames / elapsed
print(f" - Processed {num_frames} frames in {elapsed:.2f} seconds. ({fps:.2f} fps)")
return fps
.. GENERATED FROM PYTHON SOURCE LINES 489-490
The following function implements the software decoder test case.
.. GENERATED FROM PYTHON SOURCE LINES 490-510
.. code-block:: default
def test_decode_cpu(src, threads, decoder=None, frames_per_chunk=5):
s = StreamReader(src)
s.add_video_stream(frames_per_chunk, decoder=decoder, decoder_option={"threads": f"{threads}"})
num_frames = 0
device = torch.device("cuda")
t0 = time.monotonic()
for i, (chunk,) in enumerate(s.stream()):
if i == 0:
print(f" - Shape: {chunk.shape}")
num_frames += chunk.shape[0]
chunk = chunk.to(device)
elapsed = time.monotonic() - t0
fps = num_frames / elapsed
print(f" - Processed {num_frames} frames in {elapsed:.2f} seconds. ({fps:.2f} fps)")
return fps
.. GENERATED FROM PYTHON SOURCE LINES 511-513
For each resolution of video, we run multiple software decoder test
cases with different number of threads.
.. GENERATED FROM PYTHON SOURCE LINES 513-526
.. code-block:: default
def run_decode_tests(src, frames_per_chunk=5):
fps = []
print(f"Testing: {os.path.basename(src)}")
for threads in [1, 4, 8, 16]:
print(f"* Software decoding (num_threads={threads})")
fps.append(test_decode_cpu(src, threads))
print("* Hardware decoding")
fps.append(test_decode_cuda(src, decoder="h264_cuvid"))
return fps
.. GENERATED FROM PYTHON SOURCE LINES 527-531
Now we run the tests with videos of different resolutions.
QVGA
----
.. GENERATED FROM PYTHON SOURCE LINES 531-535
.. code-block:: default
src_qvga = torchaudio.utils.download_asset("tutorial-assets/testsrc2_qvga.h264.mp4")
fps_qvga = run_decode_tests(src_qvga)
.. rst-class:: sphx-glr-script-out
.. code-block:: none
0%| | 0.00/1.06M [00:00<?, ?B/s]
100%|##########| 1.06M/1.06M [00:00<00:00, 147MB/s]
Testing: testsrc2_qvga.h264.mp4
* Software decoding (num_threads=1)
- Shape: torch.Size([5, 3, 240, 320])
- Processed 900 frames in 0.50 seconds. (1814.82 fps)
* Software decoding (num_threads=4)
- Shape: torch.Size([5, 3, 240, 320])
- Processed 900 frames in 0.34 seconds. (2679.88 fps)
* Software decoding (num_threads=8)
- Shape: torch.Size([5, 3, 240, 320])
- Processed 900 frames in 0.34 seconds. (2674.27 fps)
* Software decoding (num_threads=16)
- Shape: torch.Size([5, 3, 240, 320])
- Processed 895 frames in 0.43 seconds. (2088.70 fps)
* Hardware decoding
- Shape: torch.Size([5, 3, 240, 320])
- Processed 900 frames in 2.01 seconds. (447.36 fps)
.. GENERATED FROM PYTHON SOURCE LINES 536-538
VGA
---
.. GENERATED FROM PYTHON SOURCE LINES 538-542
.. code-block:: default
src_vga = torchaudio.utils.download_asset("tutorial-assets/testsrc2_vga.h264.mp4")
fps_vga = run_decode_tests(src_vga)
.. rst-class:: sphx-glr-script-out
.. code-block:: none
0%| | 0.00/3.59M [00:00<?, ?B/s]
59%|#####9 | 2.12M/3.59M [00:00<00:00, 10.0MB/s]
100%|##########| 3.59M/3.59M [00:00<00:00, 16.3MB/s]
Testing: testsrc2_vga.h264.mp4
* Software decoding (num_threads=1)
- Shape: torch.Size([5, 3, 480, 640])
- Processed 900 frames in 1.20 seconds. (749.76 fps)
* Software decoding (num_threads=4)
- Shape: torch.Size([5, 3, 480, 640])
- Processed 900 frames in 0.71 seconds. (1274.24 fps)
* Software decoding (num_threads=8)
- Shape: torch.Size([5, 3, 480, 640])
- Processed 900 frames in 0.70 seconds. (1285.18 fps)
* Software decoding (num_threads=16)
- Shape: torch.Size([5, 3, 480, 640])
- Processed 895 frames in 0.64 seconds. (1402.77 fps)
* Hardware decoding
- Shape: torch.Size([5, 3, 480, 640])
- Processed 900 frames in 0.34 seconds. (2639.80 fps)
.. GENERATED FROM PYTHON SOURCE LINES 543-545
XGA
---
.. GENERATED FROM PYTHON SOURCE LINES 545-550
.. code-block:: default
src_xga = torchaudio.utils.download_asset("tutorial-assets/testsrc2_xga.h264.mp4")
fps_xga = run_decode_tests(src_xga)
.. rst-class:: sphx-glr-script-out
.. code-block:: none
0%| | 0.00/9.22M [00:00<?, ?B/s]
98%|#########7| 9.00M/9.22M [00:00<00:00, 35.8MB/s]
100%|##########| 9.22M/9.22M [00:00<00:00, 36.4MB/s]
Testing: testsrc2_xga.h264.mp4
* Software decoding (num_threads=1)
- Shape: torch.Size([5, 3, 768, 1024])
- Processed 900 frames in 2.70 seconds. (333.73 fps)
* Software decoding (num_threads=4)
- Shape: torch.Size([5, 3, 768, 1024])
- Processed 900 frames in 1.38 seconds. (652.84 fps)
* Software decoding (num_threads=8)
- Shape: torch.Size([5, 3, 768, 1024])
- Processed 900 frames in 1.28 seconds. (703.55 fps)
* Software decoding (num_threads=16)
- Shape: torch.Size([5, 3, 768, 1024])
- Processed 895 frames in 1.30 seconds. (690.26 fps)
* Hardware decoding
- Shape: torch.Size([5, 3, 768, 1024])
- Processed 900 frames in 0.61 seconds. (1473.92 fps)
.. GENERATED FROM PYTHON SOURCE LINES 551-555
Result
------
Now we plot the result.
.. GENERATED FROM PYTHON SOURCE LINES 555-580
.. code-block:: default
def plot():
fig, ax = plt.subplots(figsize=[9.6, 6.4])
for items in zip(fps_qvga, fps_vga, fps_xga, "ov^sx"):
ax.plot(items[:-1], marker=items[-1])
ax.grid(axis="both")
ax.set_xticks([0, 1, 2], ["QVGA (320x240)", "VGA (640x480)", "XGA (1024x768)"])
ax.legend(
[
"Software Decoding (threads=1)",
"Software Decoding (threads=4)",
"Software Decoding (threads=8)",
"Software Decoding (threads=16)",
"Hardware Decoding (CUDA Tensor)",
]
)
ax.set_title("Speed of processing video frames")
ax.set_ylabel("Frames per second")
plt.tight_layout()
plot()
.. image-sg:: /tutorials/images/sphx_glr_nvdec_tutorial_004.png
:alt: Speed of processing video frames
:srcset: /tutorials/images/sphx_glr_nvdec_tutorial_004.png
:class: sphx-glr-single-img
.. GENERATED FROM PYTHON SOURCE LINES 581-599
We observe couple of things
- Increasing the number of threads in software decoding makes the
pipeline faster, but the performance saturates around 8 threads.
- The performance gain from using hardware decoder depends on the
resolution of video.
- At lower resolutions like QVGA, hardware decoding is slower than
software decoding
- At higher resolutions like XGA, hardware decoding is faster
than software decoding.
It is worth noting that the performance gain also depends on the
type of GPU.
We observed that when decoding VGA videos using V100 or A100 GPUs,
hardware decoders are slower than software decoders. But using A10
GPU hardware deocder is faster than software decodr.
.. GENERATED FROM PYTHON SOURCE LINES 602-626
Decode and resize
-----------------
Next, we add resize operation to the pipeline.
We will compare the following pipelines.
1. Decode video using software decoder and read the frames as
PyTorch Tensor. Resize the tensor using
:py:func:`torch.nn.functional.interpolate`, then send
the resulting tensor to CUDA device.
2. Decode video using software decoder, resize the frame with
FFmpeg's filter graph, read the resized frames as PyTorch tensor,
then send it to CUDA device.
3. Decode and resize video simulaneously with HW decoder, read the
resulting frames as CUDA tensor.
The pipeline 1 represents common video loading implementations.
The pipeline 2 uses FFmpeg's filter graph, which allows to manipulate
raw frames before converting them to Tensors.
The pipeline 3 has the minimum amount of data transfer from CPU to
CUDA, which significantly contribute to performant data loading.
.. GENERATED FROM PYTHON SOURCE LINES 629-634
The following function implements the pipeline 1. It uses PyTorch's
:py:func:`torch.nn.functional.interpolate`.
We use ``bincubic`` mode, as we saw that the resulting frames are
closest to NVDEC resizing.
.. GENERATED FROM PYTHON SOURCE LINES 634-655
.. code-block:: default
def test_decode_then_resize(src, height, width, mode="bicubic", frames_per_chunk=5):
s = StreamReader(src)
s.add_video_stream(frames_per_chunk, decoder_option={"threads": "8"})
num_frames = 0
device = torch.device("cuda")
chunk = None
t0 = time.monotonic()
for (chunk,) in s.stream():
num_frames += chunk.shape[0]
chunk = torch.nn.functional.interpolate(chunk, [height, width], mode=mode, antialias=True)
chunk = chunk.to(device)
elapsed = time.monotonic() - t0
fps = num_frames / elapsed
print(f" - Shape: {chunk.shape}")
print(f" - Processed {num_frames} frames in {elapsed:.2f} seconds. ({fps:.2f} fps)")
return fps
.. GENERATED FROM PYTHON SOURCE LINES 656-662
The following function implements the pipeline 2. Frames are resized
as part of decoding process, then sent to CUDA device.
We use ``bincubic`` mode, to make the result comparable with
PyTorch-based implementation above.
.. GENERATED FROM PYTHON SOURCE LINES 662-684
.. code-block:: default
def test_decode_and_resize(src, height, width, mode="bicubic", frames_per_chunk=5):
s = StreamReader(src)
s.add_video_stream(
frames_per_chunk, filter_desc=f"scale={width}:{height}:sws_flags={mode}", decoder_option={"threads": "8"}
)
num_frames = 0
device = torch.device("cuda")
chunk = None
t0 = time.monotonic()
for (chunk,) in s.stream():
num_frames += chunk.shape[0]
chunk = chunk.to(device)
elapsed = time.monotonic() - t0
fps = num_frames / elapsed
print(f" - Shape: {chunk.shape}")
print(f" - Processed {num_frames} frames in {elapsed:.2f} seconds. ({fps:.2f} fps)")
return fps
.. GENERATED FROM PYTHON SOURCE LINES 685-687
The following function implements the pipeline 3. Resizing is
performed by NVDEC and the resulting tensor is placed on CUDA memory.
.. GENERATED FROM PYTHON SOURCE LINES 687-705
.. code-block:: default
def test_hw_decode_and_resize(src, decoder, decoder_option, hw_accel="cuda", frames_per_chunk=5):
s = StreamReader(src)
s.add_video_stream(5, decoder=decoder, decoder_option=decoder_option, hw_accel=hw_accel)
num_frames = 0
chunk = None
t0 = time.monotonic()
for (chunk,) in s.stream():
num_frames += chunk.shape[0]
elapsed = time.monotonic() - t0
fps = num_frames / elapsed
print(f" - Shape: {chunk.shape}")
print(f" - Processed {num_frames} frames in {elapsed:.2f} seconds. ({fps:.2f} fps)")
return fps
.. GENERATED FROM PYTHON SOURCE LINES 706-708
The following function run the benchmark functions on given sources.
.. GENERATED FROM PYTHON SOURCE LINES 709-723
.. code-block:: default
def run_resize_tests(src):
print(f"Testing: {os.path.basename(src)}")
height, width = 224, 224
print("* Software decoding with PyTorch interpolate")
cpu_resize1 = test_decode_then_resize(src, height=height, width=width)
print("* Software decoding with FFmpeg scale")
cpu_resize2 = test_decode_and_resize(src, height=height, width=width)
print("* Hardware decoding with resize")
cuda_resize = test_hw_decode_and_resize(src, decoder="h264_cuvid", decoder_option={"resize": f"{width}x{height}"})
return [cpu_resize1, cpu_resize2, cuda_resize]
.. GENERATED FROM PYTHON SOURCE LINES 724-725
Now we run the tests.
.. GENERATED FROM PYTHON SOURCE LINES 728-730
QVGA
----
.. GENERATED FROM PYTHON SOURCE LINES 730-733
.. code-block:: default
fps_qvga = run_resize_tests(src_qvga)
.. rst-class:: sphx-glr-script-out
.. code-block:: none
Testing: testsrc2_qvga.h264.mp4
* Software decoding with PyTorch interpolate
- Shape: torch.Size([5, 3, 224, 224])
- Processed 900 frames in 0.61 seconds. (1486.29 fps)
* Software decoding with FFmpeg scale
- Shape: torch.Size([5, 3, 224, 224])
- Processed 900 frames in 0.40 seconds. (2229.01 fps)
* Hardware decoding with resize
- Shape: torch.Size([5, 3, 224, 224])
- Processed 900 frames in 2.02 seconds. (444.56 fps)
.. GENERATED FROM PYTHON SOURCE LINES 734-736
VGA
---
.. GENERATED FROM PYTHON SOURCE LINES 736-739
.. code-block:: default
fps_vga = run_resize_tests(src_vga)
.. rst-class:: sphx-glr-script-out
.. code-block:: none
Testing: testsrc2_vga.h264.mp4
* Software decoding with PyTorch interpolate
- Shape: torch.Size([5, 3, 224, 224])
- Processed 900 frames in 1.45 seconds. (620.26 fps)
* Software decoding with FFmpeg scale
- Shape: torch.Size([5, 3, 224, 224])
- Processed 900 frames in 0.69 seconds. (1300.24 fps)
* Hardware decoding with resize
- Shape: torch.Size([5, 3, 224, 224])
- Processed 900 frames in 0.34 seconds. (2653.73 fps)
.. GENERATED FROM PYTHON SOURCE LINES 740-742
XGA
---
.. GENERATED FROM PYTHON SOURCE LINES 742-745
.. code-block:: default
fps_xga = run_resize_tests(src_xga)
.. rst-class:: sphx-glr-script-out
.. code-block:: none
Testing: testsrc2_xga.h264.mp4
* Software decoding with PyTorch interpolate
- Shape: torch.Size([5, 3, 224, 224])
- Processed 900 frames in 2.69 seconds. (334.90 fps)
* Software decoding with FFmpeg scale
- Shape: torch.Size([5, 3, 224, 224])
- Processed 900 frames in 1.06 seconds. (850.30 fps)
* Hardware decoding with resize
- Shape: torch.Size([5, 3, 224, 224])
- Processed 900 frames in 0.61 seconds. (1476.55 fps)
.. GENERATED FROM PYTHON SOURCE LINES 746-750
Result
------
Now we plot the result.
.. GENERATED FROM PYTHON SOURCE LINES 750-774
.. code-block:: default
def plot():
fig, ax = plt.subplots(figsize=[9.6, 6.4])
for items in zip(fps_qvga, fps_vga, fps_xga, "ov^sx"):
ax.plot(items[:-1], marker=items[-1])
ax.grid(axis="both")
ax.set_xticks([0, 1, 2], ["QVGA (320x240)", "VGA (640x480)", "XGA (1024x768)"])
ax.legend(
[
"Software decoding\nwith resize\n(PyTorch interpolate)",
"Software decoding\nwith resize\n(FFmpeg scale)",
"NVDEC\nwith resizing",
]
)
ax.set_title("Speed of processing video frames")
ax.set_xlabel("Input video resolution")
ax.set_ylabel("Frames per second")
plt.tight_layout()
plot()
.. image-sg:: /tutorials/images/sphx_glr_nvdec_tutorial_005.png
:alt: Speed of processing video frames
:srcset: /tutorials/images/sphx_glr_nvdec_tutorial_005.png
:class: sphx-glr-single-img
.. GENERATED FROM PYTHON SOURCE LINES 775-787
Hardware deocder shows a similar trend as previous experiment.
In fact, the performance is almost the same. Hardware resizing has
almost zero overhead for scaling down the frames.
Software decoding also shows a similar trend. Performing resizing as
part of decoding is faster. One possible explanation is that, video
frames are internally stored as YUV420P, which has half the number
of pixels compared to RGB24, or YUV444P. This means that if resizing
before copying frame data to PyTorch tensor, the number of pixels
manipulated and copied are smaller than the case where applying
resizing after frames are converted to Tensor.
.. GENERATED FROM PYTHON SOURCE LINES 790-791
Tag: :obj:`torchaudio.io`
.. rst-class:: sphx-glr-timing
**Total running time of the script:** ( 0 minutes 31.872 seconds)
.. _sphx_glr_download_tutorials_nvdec_tutorial.py:
.. only:: html
.. container:: sphx-glr-footer sphx-glr-footer-example
.. container:: sphx-glr-download sphx-glr-download-python
:download:`Download Python source code: nvdec_tutorial.py <nvdec_tutorial.py>`
.. container:: sphx-glr-download sphx-glr-download-jupyter
:download:`Download Jupyter notebook: nvdec_tutorial.ipynb <nvdec_tutorial.ipynb>`
.. only:: html
.. rst-class:: sphx-glr-signature
`Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_