.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "tutorials/streamwriter_advanced.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_tutorials_streamwriter_advanced.py: StreamWriter Advanced Usage =========================== **Author**: `Moto Hira `__ This tutorial shows how to use :py:class:`torchaudio.io.StreamWriter` to play audio and video. .. GENERATED FROM PYTHON SOURCE LINES 13-20 .. note:: This tutorial uses hardware devices, thus it is not portable across different operating systems. The tutorial was written and tested on MacBook Pro (M1, 2020). .. GENERATED FROM PYTHON SOURCE LINES 23-29 .. note:: This tutorial requires FFmpeg libraries. Please refer to :ref:`FFmpeg dependency ` for the detail. .. GENERATED FROM PYTHON SOURCE LINES 32-48 .. warning:: TorchAudio dynamically loads compatible FFmpeg libraries installed on the system. The types of supported formats (media format, encoder, encoder options etc) depend on the libraries. To check the available devices, muxers and encoders, you can use the following commands .. code-block:: console ffmpeg -muxers ffmpeg -encoders ffmpeg -devices ffmpeg -protocols .. GENERATED FROM PYTHON SOURCE LINES 51-53 Preparation ----------- .. GENERATED FROM PYTHON SOURCE LINES 54-63 .. code-block:: default import torch import torchaudio print(torch.__version__) print(torchaudio.__version__) from torchaudio.io import StreamWriter .. GENERATED FROM PYTHON SOURCE LINES 65-73 .. code-block:: default from torchaudio.utils import download_asset AUDIO_PATH = download_asset("tutorial-assets/Lab41-SRI-VOiCES-src-sp0307-ch127535-sg0042.wav") VIDEO_PATH = download_asset( "tutorial-assets/stream-api/NASAs_Most_Scientifically_Complex_Space_Observatory_Requires_Precision-MP4_small.mp4" ) .. GENERATED FROM PYTHON SOURCE LINES 74-109 Device Availability ------------------- ``StreamWriter`` takes advantage of FFmpeg's IO abstraction and writes the data to media devices such as speakers and GUI. To write to devices, provide ``format`` option to the constructor of ``StreamWriter``. Different OS will have different device options and their availabilities depend on the actual installation of FFmpeg. To check which device is available, you can use `ffmpeg -devices` command. "audiotoolbox" (speaker) and "sdl" (video GUI) are available. .. code-block:: console $ ffmpeg -devices ... Devices: D. = Demuxing supported .E = Muxing supported -- E audiotoolbox AudioToolbox output device D avfoundation AVFoundation input device D lavfi Libavfilter virtual input device E opengl OpenGL output E sdl,sdl2 SDL2 output device For details about what devices are available on which OS, please check the official FFmpeg documentation. https://ffmpeg.org/ffmpeg-devices.html .. GENERATED FROM PYTHON SOURCE LINES 112-118 Playing audio ------------- By providing ``format="audiotoolbox"`` option, the StreamWriter writes data to speaker device. .. GENERATED FROM PYTHON SOURCE LINES 119-124 .. code-block:: default # Prepare sample audio waveform, sample_rate = torchaudio.load(AUDIO_PATH, channels_first=False, normalize=False) num_frames, num_channels = waveform.shape .. GENERATED FROM PYTHON SOURCE LINES 126-131 .. code-block:: default # Configure StreamWriter to write to speaker device s = StreamWriter(dst="-", format="audiotoolbox") s.add_audio_stream(sample_rate, num_channels, format="s16") .. GENERATED FROM PYTHON SOURCE LINES 133-139 .. code-block:: default # Write audio to the device with s.open(): for i in range(0, num_frames, 256): s.write_audio_chunk(0, waveform[i : i + 256]) .. GENERATED FROM PYTHON SOURCE LINES 140-156 .. note:: Writing to "audiotoolbox" is blocking operation, but it will not wait for the aduio playback. The device must be kept open while audio is being played. The following code will close the device as soon as the audio is written and before the playback is completed. Adding :py:func:`time.sleep` will help keep the device open until the playback is completed. .. code-block:: with s.open(): s.write_audio_chunk(0, waveform) .. GENERATED FROM PYTHON SOURCE LINES 159-168 Playing Video ------------- To play video, you can use ``format="sdl"`` or ``format="opengl"``. Again, you need a version of FFmpeg with corresponding integration enabled. The available devices can be checked with ``ffmpeg -devices``. Here, we use SDL device (https://ffmpeg.org/ffmpeg-devices.html#sdl). .. GENERATED FROM PYTHON SOURCE LINES 169-177 .. code-block:: default # note: # SDL device does not support specifying frame rate, and it has to # match the refresh rate of display. frame_rate = 120 width, height = 640, 360 .. GENERATED FROM PYTHON SOURCE LINES 178-180 For we define a helper function that delegates the video loading to a background thread and give chunks .. GENERATED FROM PYTHON SOURCE LINES 181-215 .. code-block:: default running = True def video_streamer(path, frames_per_chunk): import queue import threading from torchaudio.io import StreamReader q = queue.Queue() # Streaming process that runs in background thread def _streamer(): streamer = StreamReader(path) streamer.add_basic_video_stream( frames_per_chunk, format="rgb24", frame_rate=frame_rate, width=width, height=height ) for (chunk_,) in streamer.stream(): q.put(chunk_) if not running: break # Start the background thread and fetch chunks t = threading.Thread(target=_streamer) t.start() while running: try: yield q.get() except queue.Empty: break t.join() .. GENERATED FROM PYTHON SOURCE LINES 216-222 Now we start streaming. Pressing "Q" will stop the video. .. note:: `write_video_chunk` call against SDL device blocks until SDL finishes playing the video. .. GENERATED FROM PYTHON SOURCE LINES 223-239 .. code-block:: default # Set output device to SDL s = StreamWriter("-", format="sdl") # Configure video stream (RGB24) s.add_video_stream(frame_rate, width, height, format="rgb24", encoder_format="rgb24") # Play the video with s.open(): for chunk in video_streamer(VIDEO_PATH, frames_per_chunk=256): try: s.write_video_chunk(0, chunk) except RuntimeError: running = False break .. GENERATED FROM PYTHON SOURCE LINES 240-248 .. raw:: html [`code `__] .. GENERATED FROM PYTHON SOURCE LINES 251-258 Streaming Video --------------- So far, we looked at how to write to hardware devices. There are some alternative methods for video streaming. .. GENERATED FROM PYTHON SOURCE LINES 261-294 RTMP (Real-Time Messaging Protocol) ----------------------------------- Using RMTP, you can stream media (video and/or audio) to a single client. This does not require a hardware device, but it requires a separate player. To use RMTP, specify the protocol and route in ``dst`` argument in StreamWriter constructor, then pass ``{"listen": "1"}`` option when opening the destination. StreamWriter will listen to the port and wait for a client to request the video. The call to ``open`` is blocked until a request is received. .. code-block:: s = StreamWriter(dst="rtmp://localhost:1935/live/app", format="flv") s.add_audio_stream(sample_rate=sample_rate, num_channels=num_channels, encoder="aac") s.add_video_stream(frame_rate=frame_rate, width=width, height=height) with s.open(option={"listen": "1"}): for video_chunk, audio_chunk in generator(): s.write_audio_chunk(0, audio_chunk) s.write_video_chunk(1, video_chunk) .. raw:: html [`code `__] .. GENERATED FROM PYTHON SOURCE LINES 298-326 UDP (User Datagram Protocol) ---------------------------- Using UDP, you can stream media (video and/or audio) to socket. This does not require a hardware device, but it requires a separate player. Unlike RTMP streaming and client processes are disconnected. The streaming process are not aware of client process. .. code-block:: s = StreamWriter(dst="udp://localhost:48550", format="mpegts") s.add_audio_stream(sample_rate=sample_rate, num_channels=num_channels, encoder="aac") s.add_video_stream(frame_rate=frame_rate, width=width, height=height) with s.open(): for video_chunk, audio_chunk in generator(): s.write_audio_chunk(0, audio_chunk) s.write_video_chunk(1, video_chunk) .. raw:: html [`code `__] .. GENERATED FROM PYTHON SOURCE LINES 329-330 Tag: :obj:`torchaudio.io` .. rst-class:: sphx-glr-timing **Total running time of the script:** ( 0 minutes 0.000 seconds) .. _sphx_glr_download_tutorials_streamwriter_advanced.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: streamwriter_advanced.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: streamwriter_advanced.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_