.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "tutorials/nvdec_tutorial.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here <sphx_glr_download_tutorials_nvdec_tutorial.py>` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_tutorials_nvdec_tutorial.py: Accelerated video decoding with NVDEC ===================================== .. _nvdec_tutorial: **Author**: `Moto Hira <moto@meta.com>`__ This tutorial shows how to use NVIDIA’s hardware video decoder (NVDEC) with TorchAudio, and how it improves the performance of video decoding. .. GENERATED FROM PYTHON SOURCE LINES 14-23 .. note:: This tutorial requires FFmpeg libraries compiled with HW acceleration enabled. Please refer to :ref:`Enabling GPU video decoder/encoder <enabling_hw_decoder>` for how to build FFmpeg with HW acceleration. .. GENERATED FROM PYTHON SOURCE LINES 24-31 .. code-block:: default import torch import torchaudio print(torch.__version__) print(torchaudio.__version__) .. rst-class:: sphx-glr-script-out .. code-block:: none 2.6.0 2.6.0 .. GENERATED FROM PYTHON SOURCE LINES 33-39 .. code-block:: default import os import time import matplotlib.pyplot as plt from torchaudio.io import StreamReader .. GENERATED FROM PYTHON SOURCE LINES 40-46 Check the prerequisites ----------------------- First, we check that TorchAudio correctly detects FFmpeg libraries that support HW decoder/encoder. .. GENERATED FROM PYTHON SOURCE LINES 47-50 .. code-block:: default from torchaudio.utils import ffmpeg_utils .. GENERATED FROM PYTHON SOURCE LINES 52-57 .. code-block:: default print("FFmpeg Library versions:") for k, ver in ffmpeg_utils.get_versions().items(): print(f" {k}:\t{'.'.join(str(v) for v in ver)}") .. rst-class:: sphx-glr-script-out .. code-block:: none FFmpeg Library versions: libavcodec: 60.3.100 libavdevice: 60.1.100 libavfilter: 9.3.100 libavformat: 60.3.100 libavutil: 58.2.100 .. GENERATED FROM PYTHON SOURCE LINES 59-65 .. code-block:: default print("Available NVDEC Decoders:") for k in ffmpeg_utils.get_video_decoders().keys(): if "cuvid" in k: print(f" - {k}") .. rst-class:: sphx-glr-script-out .. code-block:: none Available NVDEC Decoders: - av1_cuvid - h264_cuvid - hevc_cuvid - mjpeg_cuvid - mpeg1_cuvid - mpeg2_cuvid - mpeg4_cuvid - vc1_cuvid - vp8_cuvid - vp9_cuvid .. GENERATED FROM PYTHON SOURCE LINES 67-71 .. code-block:: default print("Avaialbe GPU:") print(torch.cuda.get_device_properties(0)) .. rst-class:: sphx-glr-script-out .. code-block:: none Avaialbe GPU: _CudaDeviceProperties(name='NVIDIA A10G', major=8, minor=6, total_memory=22502MB, multi_processor_count=80, uuid=3a6a8555-efc9-d0dc-972b-36624af6fad8, L2_cache_size=6MB) .. GENERATED FROM PYTHON SOURCE LINES 72-84 We will use the following video which has the following properties; - Codec: H.264 - Resolution: 960x540 - FPS: 29.97 - Pixel format: YUV420P .. raw:: html <video style="max-width: 100%" controls> <source src="https://download.pytorch.org/torchaudio/tutorial-assets/stream-api/NASAs_Most_Scientifically_Complex_Space_Observatory_Requires_Precision-MP4_small.mp4" type="video/mp4"> </video> .. GENERATED FROM PYTHON SOURCE LINES 88-93 .. code-block:: default src = torchaudio.utils.download_asset( "tutorial-assets/stream-api/NASAs_Most_Scientifically_Complex_Space_Observatory_Requires_Precision-MP4_small.mp4" ) .. rst-class:: sphx-glr-script-out .. code-block:: none 0%| | 0.00/31.8M [00:00<?, ?B/s] 100%|##########| 31.8M/31.8M [00:00<00:00, 545MB/s] .. GENERATED FROM PYTHON SOURCE LINES 94-101 Decoding videos with NVDEC -------------------------- To use HW video decoder, you need to specify the HW decoder when defining the output video stream by passing ``decoder`` option to :py:meth:`~torchaudio.io.StreamReader.add_video_stream` method. .. GENERATED FROM PYTHON SOURCE LINES 101-107 .. code-block:: default s = StreamReader(src) s.add_video_stream(5, decoder="h264_cuvid") s.fill_buffer() (video,) = s.pop_chunks() .. GENERATED FROM PYTHON SOURCE LINES 108-109 The video frames are decoded and returned as tensor of NCHW format. .. GENERATED FROM PYTHON SOURCE LINES 110-113 .. code-block:: default print(video.shape, video.dtype) .. rst-class:: sphx-glr-script-out .. code-block:: none torch.Size([5, 3, 540, 960]) torch.uint8 .. GENERATED FROM PYTHON SOURCE LINES 114-116 By default, the decoded frames are sent back to CPU memory, and CPU tensors are created. .. GENERATED FROM PYTHON SOURCE LINES 117-120 .. code-block:: default print(video.device) .. rst-class:: sphx-glr-script-out .. code-block:: none cpu .. GENERATED FROM PYTHON SOURCE LINES 121-134 By specifying ``hw_accel`` option, you can convert the decoded frames to CUDA tensor. ``hw_accel`` option takes string values and pass it to :py:class:`torch.device`. .. note:: Currently, ``hw_accel`` option and :py:meth:`~torchaudio.io.StreamReader.add_basic_video_stream` are not compatible. ``add_basic_video_stream`` adds post-decoding process, which is designed for frames in CPU memory. Please use :py:meth:`~torchaudio.io.StreamReader.add_video_stream`. .. GENERATED FROM PYTHON SOURCE LINES 135-144 .. code-block:: default s = StreamReader(src) s.add_video_stream(5, decoder="h264_cuvid", hw_accel="cuda:0") s.fill_buffer() (video,) = s.pop_chunks() print(video.shape, video.dtype, video.device) .. rst-class:: sphx-glr-script-out .. code-block:: none torch.Size([5, 3, 540, 960]) torch.uint8 cuda:0 .. GENERATED FROM PYTHON SOURCE LINES 145-180 .. note:: When there are multiple of GPUs available, ``StreamReader`` by default uses the first GPU. You can change this by providing ``"gpu"`` option. .. code:: # Video data is sent to CUDA device 0, decoded and # converted on the same device. s.add_video_stream( ..., decoder="h264_cuvid", decoder_option={"gpu": "0"}, hw_accel="cuda:0", ) .. note:: ``"gpu"`` option and ``hw_accel`` option can be specified independently. If they do not match, decoded frames are transfered to the device specified by ``hw_accell`` automatically. .. code:: # Video data is sent to CUDA device 0, and decoded there. # Then it is transfered to CUDA device 1, and converted to # CUDA tensor. s.add_video_stream( ..., decoder="h264_cuvid", decoder_option={"gpu": "0"}, hw_accel="cuda:1", ) .. GENERATED FROM PYTHON SOURCE LINES 182-190 Visualization ------------- Let's look at the frames decoded by HW decoder and compare them against equivalent results from software decoders. The following function seeks into the given timestamp and decode one frame with the specificed decoder. .. GENERATED FROM PYTHON SOURCE LINES 190-201 .. code-block:: default def test_decode(decoder: str, seek: float): s = StreamReader(src) s.seek(seek) s.add_video_stream(1, decoder=decoder) s.fill_buffer() (video,) = s.pop_chunks() return video[0] .. GENERATED FROM PYTHON SOURCE LINES 203-209 .. code-block:: default timestamps = [12, 19, 45, 131, 180] cpu_frames = [test_decode(decoder="h264", seek=ts) for ts in timestamps] cuda_frames = [test_decode(decoder="h264_cuvid", seek=ts) for ts in timestamps] .. GENERATED FROM PYTHON SOURCE LINES 210-216 .. note:: Currently, HW decoder does not support colorspace conversion. Decoded frames are YUV format. The following function performs YUV to RGB covnersion (and axis shuffling for plotting). .. GENERATED FROM PYTHON SOURCE LINES 217-238 .. code-block:: default def yuv_to_rgb(frames): frames = frames.cpu().to(torch.float) y = frames[..., 0, :, :] u = frames[..., 1, :, :] v = frames[..., 2, :, :] y /= 255 u = u / 255 - 0.5 v = v / 255 - 0.5 r = y + 1.14 * v g = y + -0.396 * u - 0.581 * v b = y + 2.029 * u rgb = torch.stack([r, g, b], -1) rgb = (rgb * 255).clamp(0, 255).to(torch.uint8) return rgb.numpy() .. GENERATED FROM PYTHON SOURCE LINES 239-241 Now we visualize the resutls. .. GENERATED FROM PYTHON SOURCE LINES 242-259 .. code-block:: default def plot(): n_rows = len(timestamps) fig, axes = plt.subplots(n_rows, 2, figsize=[12.8, 16.0]) for i in range(n_rows): axes[i][0].imshow(yuv_to_rgb(cpu_frames[i])) axes[i][1].imshow(yuv_to_rgb(cuda_frames[i])) axes[0][0].set_title("Software decoder") axes[0][1].set_title("HW decoder") plt.setp(axes, xticks=[], yticks=[]) plt.tight_layout() plot() .. image-sg:: /tutorials/images/sphx_glr_nvdec_tutorial_001.png :alt: Software decoder, HW decoder :srcset: /tutorials/images/sphx_glr_nvdec_tutorial_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 260-263 They are indistinguishable to the eyes of the author. Feel free to let us know if you spot something. :) .. GENERATED FROM PYTHON SOURCE LINES 267-285 HW resizing and cropping ------------------------ You can use ``decoder_option`` argument to provide decoder-specific options. The following options are often relevant in preprocessing. - ``resize``: Resize the frame into ``(width)x(height)``. - ``crop``: Crop the frame ``(top)x(bottom)x(left)x(right)``. Note that the specified values are the amount of rows/columns removed. The final image size is ``(width - left - right)x(height - top -bottom)``. If ``crop`` and ``resize`` options are used together, ``crop`` is performed first. For other available options, please run ``ffmpeg -h decoder=h264_cuvid``. .. GENERATED FROM PYTHON SOURCE LINES 285-297 .. code-block:: default def test_options(option): s = StreamReader(src) s.seek(87) s.add_video_stream(1, decoder="h264_cuvid", hw_accel="cuda:0", decoder_option=option) s.fill_buffer() (video,) = s.pop_chunks() print(f"Option: {option}:\t{video.shape}") return video[0] .. GENERATED FROM PYTHON SOURCE LINES 299-306 .. code-block:: default original = test_options(option=None) resized = test_options(option={"resize": "480x270"}) cropped = test_options(option={"crop": "135x135x240x240"}) cropped_and_resized = test_options(option={"crop": "135x135x240x240", "resize": "640x360"}) .. rst-class:: sphx-glr-script-out .. code-block:: none Option: None: torch.Size([1, 3, 540, 960]) Option: {'resize': '480x270'}: torch.Size([1, 3, 270, 480]) Option: {'crop': '135x135x240x240'}: torch.Size([1, 3, 270, 480]) Option: {'crop': '135x135x240x240', 'resize': '640x360'}: torch.Size([1, 3, 360, 640]) .. GENERATED FROM PYTHON SOURCE LINES 308-327 .. code-block:: default def plot(): fig, axes = plt.subplots(2, 2, figsize=[12.8, 9.6]) axes[0][0].imshow(yuv_to_rgb(original)) axes[0][1].imshow(yuv_to_rgb(resized)) axes[1][0].imshow(yuv_to_rgb(cropped)) axes[1][1].imshow(yuv_to_rgb(cropped_and_resized)) axes[0][0].set_title("Original") axes[0][1].set_title("Resized") axes[1][0].set_title("Cropped") axes[1][1].set_title("Cropped and resized") plt.tight_layout() return fig plot() .. image-sg:: /tutorials/images/sphx_glr_nvdec_tutorial_002.png :alt: Original, Resized, Cropped, Cropped and resized :srcset: /tutorials/images/sphx_glr_nvdec_tutorial_002.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none <Figure size 1280x960 with 4 Axes> .. GENERATED FROM PYTHON SOURCE LINES 328-350 Comparing resizing methods -------------------------- Unlike software scaling, NVDEC does not provide an option to choose the scaling algorithm. In ML applicatoins, it is often necessary to construct a preprocessing pipeline with a similar numerical property. So here we compare the result of hardware resizing with software resizing of different algorithms. We will use the following video, which contains the test pattern generated using the following command. .. code:: ffmpeg -y -f lavfi -t 12.05 -i mptestsrc -movflags +faststart mptestsrc.mp4 .. raw:: html <video style="max-width: 100%" controls> <source src="https://download.pytorch.org/torchaudio/tutorial-assets/mptestsrc.mp4" type="video/mp4"> </video> .. GENERATED FROM PYTHON SOURCE LINES 354-358 .. code-block:: default test_src = torchaudio.utils.download_asset("tutorial-assets/mptestsrc.mp4") .. rst-class:: sphx-glr-script-out .. code-block:: none 0%| | 0.00/232k [00:00<?, ?B/s] 100%|##########| 232k/232k [00:00<00:00, 41.6MB/s] .. GENERATED FROM PYTHON SOURCE LINES 359-362 The following function decodes video and apply the specified scaling algorithm. .. GENERATED FROM PYTHON SOURCE LINES 362-374 .. code-block:: default def decode_resize_ffmpeg(mode, height, width, seek): filter_desc = None if mode is None else f"scale={width}:{height}:sws_flags={mode}" s = StreamReader(test_src) s.add_video_stream(1, filter_desc=filter_desc) s.seek(seek) s.fill_buffer() (chunk,) = s.pop_chunks() return chunk .. GENERATED FROM PYTHON SOURCE LINES 375-377 The following function uses HW decoder to decode video and resize. .. GENERATED FROM PYTHON SOURCE LINES 377-388 .. code-block:: default def decode_resize_cuvid(height, width, seek): s = StreamReader(test_src) s.add_video_stream(1, decoder="h264_cuvid", decoder_option={"resize": f"{width}x{height}"}, hw_accel="cuda:0") s.seek(seek) s.fill_buffer() (chunk,) = s.pop_chunks() return chunk.cpu() .. GENERATED FROM PYTHON SOURCE LINES 389-390 Now we execute them and visualize the resulting frames. .. GENERATED FROM PYTHON SOURCE LINES 390-406 .. code-block:: default params = {"height": 224, "width": 224, "seek": 3} frames = [ decode_resize_ffmpeg(None, **params), decode_resize_ffmpeg("neighbor", **params), decode_resize_ffmpeg("bilinear", **params), decode_resize_ffmpeg("bicubic", **params), decode_resize_cuvid(**params), decode_resize_ffmpeg("spline", **params), decode_resize_ffmpeg("lanczos:param0=1", **params), decode_resize_ffmpeg("lanczos:param0=3", **params), decode_resize_ffmpeg("lanczos:param0=5", **params), ] .. GENERATED FROM PYTHON SOURCE LINES 408-432 .. code-block:: default def plot(): fig, axes = plt.subplots(3, 3, figsize=[12.8, 15.2]) for i, f in enumerate(frames): h, w = f.shape[2:4] f = f[..., : h // 4, : w // 4] axes[i // 3][i % 3].imshow(yuv_to_rgb(f[0])) axes[0][0].set_title("Original") axes[0][1].set_title("nearest neighbor") axes[0][2].set_title("bilinear") axes[1][0].set_title("bicubic") axes[1][1].set_title("NVDEC") axes[1][2].set_title("spline") axes[2][0].set_title("lanczos(1)") axes[2][1].set_title("lanczos(3)") axes[2][2].set_title("lanczos(5)") plt.setp(axes, xticks=[], yticks=[]) plt.tight_layout() plot() .. image-sg:: /tutorials/images/sphx_glr_nvdec_tutorial_003.png :alt: Original, nearest neighbor, bilinear, bicubic, NVDEC, spline, lanczos(1), lanczos(3), lanczos(5) :srcset: /tutorials/images/sphx_glr_nvdec_tutorial_003.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 433-436 None of them is exactly the same. To the eyes of authors, lanczos(1) appears to be most similar to NVDEC. The bicubic looks close as well. .. GENERATED FROM PYTHON SOURCE LINES 438-444 Benchmark NVDEC with StreamReader --------------------------------- In this section, we compare the performace of software video decoding and HW video decoding. .. GENERATED FROM PYTHON SOURCE LINES 447-466 Decode as CUDA frames --------------------- First, we compare the time it takes for software decoder and hardware encoder to decode the same video. To make the result comparable, when using software decoder, we move the resulting tensor to CUDA. The procedures to test look like the following - Use hardware decoder and place data on CUDA directly - Use software decoder, generate CPU Tensors and move them to CUDA. .. note: Because HW decoder currently only supports reading videos as YUV444P format, we decode frames into YUV444P format for the case of software decoder as well. .. GENERATED FROM PYTHON SOURCE LINES 469-470 The following function implements the hardware decoder test case. .. GENERATED FROM PYTHON SOURCE LINES 470-488 .. code-block:: default def test_decode_cuda(src, decoder, hw_accel="cuda", frames_per_chunk=5): s = StreamReader(src) s.add_video_stream(frames_per_chunk, decoder=decoder, hw_accel=hw_accel) num_frames = 0 chunk = None t0 = time.monotonic() for (chunk,) in s.stream(): num_frames += chunk.shape[0] elapsed = time.monotonic() - t0 print(f" - Shape: {chunk.shape}") fps = num_frames / elapsed print(f" - Processed {num_frames} frames in {elapsed:.2f} seconds. ({fps:.2f} fps)") return fps .. GENERATED FROM PYTHON SOURCE LINES 489-490 The following function implements the software decoder test case. .. GENERATED FROM PYTHON SOURCE LINES 490-510 .. code-block:: default def test_decode_cpu(src, threads, decoder=None, frames_per_chunk=5): s = StreamReader(src) s.add_video_stream(frames_per_chunk, decoder=decoder, decoder_option={"threads": f"{threads}"}) num_frames = 0 device = torch.device("cuda") t0 = time.monotonic() for i, (chunk,) in enumerate(s.stream()): if i == 0: print(f" - Shape: {chunk.shape}") num_frames += chunk.shape[0] chunk = chunk.to(device) elapsed = time.monotonic() - t0 fps = num_frames / elapsed print(f" - Processed {num_frames} frames in {elapsed:.2f} seconds. ({fps:.2f} fps)") return fps .. GENERATED FROM PYTHON SOURCE LINES 511-513 For each resolution of video, we run multiple software decoder test cases with different number of threads. .. GENERATED FROM PYTHON SOURCE LINES 513-526 .. code-block:: default def run_decode_tests(src, frames_per_chunk=5): fps = [] print(f"Testing: {os.path.basename(src)}") for threads in [1, 4, 8, 16]: print(f"* Software decoding (num_threads={threads})") fps.append(test_decode_cpu(src, threads)) print("* Hardware decoding") fps.append(test_decode_cuda(src, decoder="h264_cuvid")) return fps .. GENERATED FROM PYTHON SOURCE LINES 527-531 Now we run the tests with videos of different resolutions. QVGA ---- .. GENERATED FROM PYTHON SOURCE LINES 531-535 .. code-block:: default src_qvga = torchaudio.utils.download_asset("tutorial-assets/testsrc2_qvga.h264.mp4") fps_qvga = run_decode_tests(src_qvga) .. rst-class:: sphx-glr-script-out .. code-block:: none 0%| | 0.00/1.06M [00:00<?, ?B/s] 100%|##########| 1.06M/1.06M [00:00<00:00, 147MB/s] Testing: testsrc2_qvga.h264.mp4 * Software decoding (num_threads=1) - Shape: torch.Size([5, 3, 240, 320]) - Processed 900 frames in 0.50 seconds. (1814.82 fps) * Software decoding (num_threads=4) - Shape: torch.Size([5, 3, 240, 320]) - Processed 900 frames in 0.34 seconds. (2679.88 fps) * Software decoding (num_threads=8) - Shape: torch.Size([5, 3, 240, 320]) - Processed 900 frames in 0.34 seconds. (2674.27 fps) * Software decoding (num_threads=16) - Shape: torch.Size([5, 3, 240, 320]) - Processed 895 frames in 0.43 seconds. (2088.70 fps) * Hardware decoding - Shape: torch.Size([5, 3, 240, 320]) - Processed 900 frames in 2.01 seconds. (447.36 fps) .. GENERATED FROM PYTHON SOURCE LINES 536-538 VGA --- .. GENERATED FROM PYTHON SOURCE LINES 538-542 .. code-block:: default src_vga = torchaudio.utils.download_asset("tutorial-assets/testsrc2_vga.h264.mp4") fps_vga = run_decode_tests(src_vga) .. rst-class:: sphx-glr-script-out .. code-block:: none 0%| | 0.00/3.59M [00:00<?, ?B/s] 59%|#####9 | 2.12M/3.59M [00:00<00:00, 10.0MB/s] 100%|##########| 3.59M/3.59M [00:00<00:00, 16.3MB/s] Testing: testsrc2_vga.h264.mp4 * Software decoding (num_threads=1) - Shape: torch.Size([5, 3, 480, 640]) - Processed 900 frames in 1.20 seconds. (749.76 fps) * Software decoding (num_threads=4) - Shape: torch.Size([5, 3, 480, 640]) - Processed 900 frames in 0.71 seconds. (1274.24 fps) * Software decoding (num_threads=8) - Shape: torch.Size([5, 3, 480, 640]) - Processed 900 frames in 0.70 seconds. (1285.18 fps) * Software decoding (num_threads=16) - Shape: torch.Size([5, 3, 480, 640]) - Processed 895 frames in 0.64 seconds. (1402.77 fps) * Hardware decoding - Shape: torch.Size([5, 3, 480, 640]) - Processed 900 frames in 0.34 seconds. (2639.80 fps) .. GENERATED FROM PYTHON SOURCE LINES 543-545 XGA --- .. GENERATED FROM PYTHON SOURCE LINES 545-550 .. code-block:: default src_xga = torchaudio.utils.download_asset("tutorial-assets/testsrc2_xga.h264.mp4") fps_xga = run_decode_tests(src_xga) .. rst-class:: sphx-glr-script-out .. code-block:: none 0%| | 0.00/9.22M [00:00<?, ?B/s] 98%|#########7| 9.00M/9.22M [00:00<00:00, 35.8MB/s] 100%|##########| 9.22M/9.22M [00:00<00:00, 36.4MB/s] Testing: testsrc2_xga.h264.mp4 * Software decoding (num_threads=1) - Shape: torch.Size([5, 3, 768, 1024]) - Processed 900 frames in 2.70 seconds. (333.73 fps) * Software decoding (num_threads=4) - Shape: torch.Size([5, 3, 768, 1024]) - Processed 900 frames in 1.38 seconds. (652.84 fps) * Software decoding (num_threads=8) - Shape: torch.Size([5, 3, 768, 1024]) - Processed 900 frames in 1.28 seconds. (703.55 fps) * Software decoding (num_threads=16) - Shape: torch.Size([5, 3, 768, 1024]) - Processed 895 frames in 1.30 seconds. (690.26 fps) * Hardware decoding - Shape: torch.Size([5, 3, 768, 1024]) - Processed 900 frames in 0.61 seconds. (1473.92 fps) .. GENERATED FROM PYTHON SOURCE LINES 551-555 Result ------ Now we plot the result. .. GENERATED FROM PYTHON SOURCE LINES 555-580 .. code-block:: default def plot(): fig, ax = plt.subplots(figsize=[9.6, 6.4]) for items in zip(fps_qvga, fps_vga, fps_xga, "ov^sx"): ax.plot(items[:-1], marker=items[-1]) ax.grid(axis="both") ax.set_xticks([0, 1, 2], ["QVGA (320x240)", "VGA (640x480)", "XGA (1024x768)"]) ax.legend( [ "Software Decoding (threads=1)", "Software Decoding (threads=4)", "Software Decoding (threads=8)", "Software Decoding (threads=16)", "Hardware Decoding (CUDA Tensor)", ] ) ax.set_title("Speed of processing video frames") ax.set_ylabel("Frames per second") plt.tight_layout() plot() .. image-sg:: /tutorials/images/sphx_glr_nvdec_tutorial_004.png :alt: Speed of processing video frames :srcset: /tutorials/images/sphx_glr_nvdec_tutorial_004.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 581-599 We observe couple of things - Increasing the number of threads in software decoding makes the pipeline faster, but the performance saturates around 8 threads. - The performance gain from using hardware decoder depends on the resolution of video. - At lower resolutions like QVGA, hardware decoding is slower than software decoding - At higher resolutions like XGA, hardware decoding is faster than software decoding. It is worth noting that the performance gain also depends on the type of GPU. We observed that when decoding VGA videos using V100 or A100 GPUs, hardware decoders are slower than software decoders. But using A10 GPU hardware deocder is faster than software decodr. .. GENERATED FROM PYTHON SOURCE LINES 602-626 Decode and resize ----------------- Next, we add resize operation to the pipeline. We will compare the following pipelines. 1. Decode video using software decoder and read the frames as PyTorch Tensor. Resize the tensor using :py:func:`torch.nn.functional.interpolate`, then send the resulting tensor to CUDA device. 2. Decode video using software decoder, resize the frame with FFmpeg's filter graph, read the resized frames as PyTorch tensor, then send it to CUDA device. 3. Decode and resize video simulaneously with HW decoder, read the resulting frames as CUDA tensor. The pipeline 1 represents common video loading implementations. The pipeline 2 uses FFmpeg's filter graph, which allows to manipulate raw frames before converting them to Tensors. The pipeline 3 has the minimum amount of data transfer from CPU to CUDA, which significantly contribute to performant data loading. .. GENERATED FROM PYTHON SOURCE LINES 629-634 The following function implements the pipeline 1. It uses PyTorch's :py:func:`torch.nn.functional.interpolate`. We use ``bincubic`` mode, as we saw that the resulting frames are closest to NVDEC resizing. .. GENERATED FROM PYTHON SOURCE LINES 634-655 .. code-block:: default def test_decode_then_resize(src, height, width, mode="bicubic", frames_per_chunk=5): s = StreamReader(src) s.add_video_stream(frames_per_chunk, decoder_option={"threads": "8"}) num_frames = 0 device = torch.device("cuda") chunk = None t0 = time.monotonic() for (chunk,) in s.stream(): num_frames += chunk.shape[0] chunk = torch.nn.functional.interpolate(chunk, [height, width], mode=mode, antialias=True) chunk = chunk.to(device) elapsed = time.monotonic() - t0 fps = num_frames / elapsed print(f" - Shape: {chunk.shape}") print(f" - Processed {num_frames} frames in {elapsed:.2f} seconds. ({fps:.2f} fps)") return fps .. GENERATED FROM PYTHON SOURCE LINES 656-662 The following function implements the pipeline 2. Frames are resized as part of decoding process, then sent to CUDA device. We use ``bincubic`` mode, to make the result comparable with PyTorch-based implementation above. .. GENERATED FROM PYTHON SOURCE LINES 662-684 .. code-block:: default def test_decode_and_resize(src, height, width, mode="bicubic", frames_per_chunk=5): s = StreamReader(src) s.add_video_stream( frames_per_chunk, filter_desc=f"scale={width}:{height}:sws_flags={mode}", decoder_option={"threads": "8"} ) num_frames = 0 device = torch.device("cuda") chunk = None t0 = time.monotonic() for (chunk,) in s.stream(): num_frames += chunk.shape[0] chunk = chunk.to(device) elapsed = time.monotonic() - t0 fps = num_frames / elapsed print(f" - Shape: {chunk.shape}") print(f" - Processed {num_frames} frames in {elapsed:.2f} seconds. ({fps:.2f} fps)") return fps .. GENERATED FROM PYTHON SOURCE LINES 685-687 The following function implements the pipeline 3. Resizing is performed by NVDEC and the resulting tensor is placed on CUDA memory. .. GENERATED FROM PYTHON SOURCE LINES 687-705 .. code-block:: default def test_hw_decode_and_resize(src, decoder, decoder_option, hw_accel="cuda", frames_per_chunk=5): s = StreamReader(src) s.add_video_stream(5, decoder=decoder, decoder_option=decoder_option, hw_accel=hw_accel) num_frames = 0 chunk = None t0 = time.monotonic() for (chunk,) in s.stream(): num_frames += chunk.shape[0] elapsed = time.monotonic() - t0 fps = num_frames / elapsed print(f" - Shape: {chunk.shape}") print(f" - Processed {num_frames} frames in {elapsed:.2f} seconds. ({fps:.2f} fps)") return fps .. GENERATED FROM PYTHON SOURCE LINES 706-708 The following function run the benchmark functions on given sources. .. GENERATED FROM PYTHON SOURCE LINES 709-723 .. code-block:: default def run_resize_tests(src): print(f"Testing: {os.path.basename(src)}") height, width = 224, 224 print("* Software decoding with PyTorch interpolate") cpu_resize1 = test_decode_then_resize(src, height=height, width=width) print("* Software decoding with FFmpeg scale") cpu_resize2 = test_decode_and_resize(src, height=height, width=width) print("* Hardware decoding with resize") cuda_resize = test_hw_decode_and_resize(src, decoder="h264_cuvid", decoder_option={"resize": f"{width}x{height}"}) return [cpu_resize1, cpu_resize2, cuda_resize] .. GENERATED FROM PYTHON SOURCE LINES 724-725 Now we run the tests. .. GENERATED FROM PYTHON SOURCE LINES 728-730 QVGA ---- .. GENERATED FROM PYTHON SOURCE LINES 730-733 .. code-block:: default fps_qvga = run_resize_tests(src_qvga) .. rst-class:: sphx-glr-script-out .. code-block:: none Testing: testsrc2_qvga.h264.mp4 * Software decoding with PyTorch interpolate - Shape: torch.Size([5, 3, 224, 224]) - Processed 900 frames in 0.61 seconds. (1486.29 fps) * Software decoding with FFmpeg scale - Shape: torch.Size([5, 3, 224, 224]) - Processed 900 frames in 0.40 seconds. (2229.01 fps) * Hardware decoding with resize - Shape: torch.Size([5, 3, 224, 224]) - Processed 900 frames in 2.02 seconds. (444.56 fps) .. GENERATED FROM PYTHON SOURCE LINES 734-736 VGA --- .. GENERATED FROM PYTHON SOURCE LINES 736-739 .. code-block:: default fps_vga = run_resize_tests(src_vga) .. rst-class:: sphx-glr-script-out .. code-block:: none Testing: testsrc2_vga.h264.mp4 * Software decoding with PyTorch interpolate - Shape: torch.Size([5, 3, 224, 224]) - Processed 900 frames in 1.45 seconds. (620.26 fps) * Software decoding with FFmpeg scale - Shape: torch.Size([5, 3, 224, 224]) - Processed 900 frames in 0.69 seconds. (1300.24 fps) * Hardware decoding with resize - Shape: torch.Size([5, 3, 224, 224]) - Processed 900 frames in 0.34 seconds. (2653.73 fps) .. GENERATED FROM PYTHON SOURCE LINES 740-742 XGA --- .. GENERATED FROM PYTHON SOURCE LINES 742-745 .. code-block:: default fps_xga = run_resize_tests(src_xga) .. rst-class:: sphx-glr-script-out .. code-block:: none Testing: testsrc2_xga.h264.mp4 * Software decoding with PyTorch interpolate - Shape: torch.Size([5, 3, 224, 224]) - Processed 900 frames in 2.69 seconds. (334.90 fps) * Software decoding with FFmpeg scale - Shape: torch.Size([5, 3, 224, 224]) - Processed 900 frames in 1.06 seconds. (850.30 fps) * Hardware decoding with resize - Shape: torch.Size([5, 3, 224, 224]) - Processed 900 frames in 0.61 seconds. (1476.55 fps) .. GENERATED FROM PYTHON SOURCE LINES 746-750 Result ------ Now we plot the result. .. GENERATED FROM PYTHON SOURCE LINES 750-774 .. code-block:: default def plot(): fig, ax = plt.subplots(figsize=[9.6, 6.4]) for items in zip(fps_qvga, fps_vga, fps_xga, "ov^sx"): ax.plot(items[:-1], marker=items[-1]) ax.grid(axis="both") ax.set_xticks([0, 1, 2], ["QVGA (320x240)", "VGA (640x480)", "XGA (1024x768)"]) ax.legend( [ "Software decoding\nwith resize\n(PyTorch interpolate)", "Software decoding\nwith resize\n(FFmpeg scale)", "NVDEC\nwith resizing", ] ) ax.set_title("Speed of processing video frames") ax.set_xlabel("Input video resolution") ax.set_ylabel("Frames per second") plt.tight_layout() plot() .. image-sg:: /tutorials/images/sphx_glr_nvdec_tutorial_005.png :alt: Speed of processing video frames :srcset: /tutorials/images/sphx_glr_nvdec_tutorial_005.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 775-787 Hardware deocder shows a similar trend as previous experiment. In fact, the performance is almost the same. Hardware resizing has almost zero overhead for scaling down the frames. Software decoding also shows a similar trend. Performing resizing as part of decoding is faster. One possible explanation is that, video frames are internally stored as YUV420P, which has half the number of pixels compared to RGB24, or YUV444P. This means that if resizing before copying frame data to PyTorch tensor, the number of pixels manipulated and copied are smaller than the case where applying resizing after frames are converted to Tensor. .. GENERATED FROM PYTHON SOURCE LINES 790-791 Tag: :obj:`torchaudio.io` .. rst-class:: sphx-glr-timing **Total running time of the script:** ( 0 minutes 31.872 seconds) .. _sphx_glr_download_tutorials_nvdec_tutorial.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: nvdec_tutorial.py <nvdec_tutorial.py>` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: nvdec_tutorial.ipynb <nvdec_tutorial.ipynb>` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_