.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "tutorials/streamwriter_advanced.py"
.. LINE NUMBERS ARE GIVEN BELOW.
.. only:: html
.. note::
:class: sphx-glr-download-link-note
Click :ref:`here <sphx_glr_download_tutorials_streamwriter_advanced.py>`
to download the full example code
.. rst-class:: sphx-glr-example-title
.. _sphx_glr_tutorials_streamwriter_advanced.py:
StreamWriter Advanced Usage
===========================
**Author**: `Moto Hira <moto@meta.com>`__
This tutorial shows how to use :py:class:`torchaudio.io.StreamWriter` to
play audio and video.
.. GENERATED FROM PYTHON SOURCE LINES 13-20
.. note::
This tutorial uses hardware devices, thus it is not portable across
different operating systems.
The tutorial was written and tested on MacBook Pro (M1, 2020).
.. GENERATED FROM PYTHON SOURCE LINES 23-29
.. note::
This tutorial requires FFmpeg libraries.
Please refer to :ref:`FFmpeg dependency <ffmpeg_dependency>` for
the detail.
.. GENERATED FROM PYTHON SOURCE LINES 32-48
.. warning::
TorchAudio dynamically loads compatible FFmpeg libraries
installed on the system.
The types of supported formats (media format, encoder, encoder
options etc) depend on the libraries.
To check the available devices, muxers and encoders, you can use the
following commands
.. code-block:: console
ffmpeg -muxers
ffmpeg -encoders
ffmpeg -devices
ffmpeg -protocols
.. GENERATED FROM PYTHON SOURCE LINES 51-53
Preparation
-----------
.. GENERATED FROM PYTHON SOURCE LINES 54-63
.. code-block:: default
import torch
import torchaudio
print(torch.__version__)
print(torchaudio.__version__)
from torchaudio.io import StreamWriter
.. GENERATED FROM PYTHON SOURCE LINES 65-73
.. code-block:: default
from torchaudio.utils import download_asset
AUDIO_PATH = download_asset("tutorial-assets/Lab41-SRI-VOiCES-src-sp0307-ch127535-sg0042.wav")
VIDEO_PATH = download_asset(
"tutorial-assets/stream-api/NASAs_Most_Scientifically_Complex_Space_Observatory_Requires_Precision-MP4_small.mp4"
)
.. GENERATED FROM PYTHON SOURCE LINES 74-109
Device Availability
-------------------
``StreamWriter`` takes advantage of FFmpeg's IO abstraction and
writes the data to media devices such as speakers and GUI.
To write to devices, provide ``format`` option to the constructor
of ``StreamWriter``.
Different OS will have different device options and their availabilities
depend on the actual installation of FFmpeg.
To check which device is available, you can use `ffmpeg -devices`
command.
"audiotoolbox" (speaker) and "sdl" (video GUI)
are available.
.. code-block:: console
$ ffmpeg -devices
...
Devices:
D. = Demuxing supported
.E = Muxing supported
--
E audiotoolbox AudioToolbox output device
D avfoundation AVFoundation input device
D lavfi Libavfilter virtual input device
E opengl OpenGL output
E sdl,sdl2 SDL2 output device
For details about what devices are available on which OS, please check
the official FFmpeg documentation. https://ffmpeg.org/ffmpeg-devices.html
.. GENERATED FROM PYTHON SOURCE LINES 112-118
Playing audio
-------------
By providing ``format="audiotoolbox"`` option, the StreamWriter writes
data to speaker device.
.. GENERATED FROM PYTHON SOURCE LINES 119-124
.. code-block:: default
# Prepare sample audio
waveform, sample_rate = torchaudio.load(AUDIO_PATH, channels_first=False, normalize=False)
num_frames, num_channels = waveform.shape
.. GENERATED FROM PYTHON SOURCE LINES 126-131
.. code-block:: default
# Configure StreamWriter to write to speaker device
s = StreamWriter(dst="-", format="audiotoolbox")
s.add_audio_stream(sample_rate, num_channels, format="s16")
.. GENERATED FROM PYTHON SOURCE LINES 133-139
.. code-block:: default
# Write audio to the device
with s.open():
for i in range(0, num_frames, 256):
s.write_audio_chunk(0, waveform[i : i + 256])
.. GENERATED FROM PYTHON SOURCE LINES 140-156
.. note::
Writing to "audiotoolbox" is blocking operation, but it will not
wait for the aduio playback. The device must be kept open while
audio is being played.
The following code will close the device as soon as the audio is
written and before the playback is completed.
Adding :py:func:`time.sleep` will help keep the device open until
the playback is completed.
.. code-block::
with s.open():
s.write_audio_chunk(0, waveform)
.. GENERATED FROM PYTHON SOURCE LINES 159-168
Playing Video
-------------
To play video, you can use ``format="sdl"`` or ``format="opengl"``.
Again, you need a version of FFmpeg with corresponding integration
enabled. The available devices can be checked with ``ffmpeg -devices``.
Here, we use SDL device (https://ffmpeg.org/ffmpeg-devices.html#sdl).
.. GENERATED FROM PYTHON SOURCE LINES 169-177
.. code-block:: default
# note:
# SDL device does not support specifying frame rate, and it has to
# match the refresh rate of display.
frame_rate = 120
width, height = 640, 360
.. GENERATED FROM PYTHON SOURCE LINES 178-180
For we define a helper function that delegates the video loading to
a background thread and give chunks
.. GENERATED FROM PYTHON SOURCE LINES 181-215
.. code-block:: default
running = True
def video_streamer(path, frames_per_chunk):
import queue
import threading
from torchaudio.io import StreamReader
q = queue.Queue()
# Streaming process that runs in background thread
def _streamer():
streamer = StreamReader(path)
streamer.add_basic_video_stream(
frames_per_chunk, format="rgb24", frame_rate=frame_rate, width=width, height=height
)
for (chunk_,) in streamer.stream():
q.put(chunk_)
if not running:
break
# Start the background thread and fetch chunks
t = threading.Thread(target=_streamer)
t.start()
while running:
try:
yield q.get()
except queue.Empty:
break
t.join()
.. GENERATED FROM PYTHON SOURCE LINES 216-222
Now we start streaming. Pressing "Q" will stop the video.
.. note::
`write_video_chunk` call against SDL device blocks until SDL finishes
playing the video.
.. GENERATED FROM PYTHON SOURCE LINES 223-239
.. code-block:: default
# Set output device to SDL
s = StreamWriter("-", format="sdl")
# Configure video stream (RGB24)
s.add_video_stream(frame_rate, width, height, format="rgb24", encoder_format="rgb24")
# Play the video
with s.open():
for chunk in video_streamer(VIDEO_PATH, frames_per_chunk=256):
try:
s.write_video_chunk(0, chunk)
except RuntimeError:
running = False
break
.. GENERATED FROM PYTHON SOURCE LINES 240-248
.. raw:: html
<video width="490px" controls autoplay loop muted>
<source src="https://download.pytorch.org/torchaudio/tutorial-assets/torchaudio-sdl-demo.mp4">
</video>
[`code <https://download.pytorch.org/torchaudio/tutorial-assets/sdl.py>`__]
.. GENERATED FROM PYTHON SOURCE LINES 251-258
Streaming Video
---------------
So far, we looked at how to write to hardware devices. There are some
alternative methods for video streaming.
.. GENERATED FROM PYTHON SOURCE LINES 261-294
RTMP (Real-Time Messaging Protocol)
-----------------------------------
Using RMTP, you can stream media (video and/or audio) to a single client.
This does not require a hardware device, but it requires a separate player.
To use RMTP, specify the protocol and route in ``dst`` argument in
StreamWriter constructor, then pass ``{"listen": "1"}`` option when opening
the destination.
StreamWriter will listen to the port and wait for a client to request the video.
The call to ``open`` is blocked until a request is received.
.. code-block::
s = StreamWriter(dst="rtmp://localhost:1935/live/app", format="flv")
s.add_audio_stream(sample_rate=sample_rate, num_channels=num_channels, encoder="aac")
s.add_video_stream(frame_rate=frame_rate, width=width, height=height)
with s.open(option={"listen": "1"}):
for video_chunk, audio_chunk in generator():
s.write_audio_chunk(0, audio_chunk)
s.write_video_chunk(1, video_chunk)
.. raw:: html
<video width="490px" controls autoplay loop muted>
<source src="https://download.pytorch.org/torchaudio/tutorial-assets/torchaudio-rtmp-demo.mp4">
</video>
[`code <https://download.pytorch.org/torchaudio/tutorial-assets/rtmp.py>`__]
.. GENERATED FROM PYTHON SOURCE LINES 298-326
UDP (User Datagram Protocol)
----------------------------
Using UDP, you can stream media (video and/or audio) to socket.
This does not require a hardware device, but it requires a separate player.
Unlike RTMP streaming and client processes are disconnected.
The streaming process are not aware of client process.
.. code-block::
s = StreamWriter(dst="udp://localhost:48550", format="mpegts")
s.add_audio_stream(sample_rate=sample_rate, num_channels=num_channels, encoder="aac")
s.add_video_stream(frame_rate=frame_rate, width=width, height=height)
with s.open():
for video_chunk, audio_chunk in generator():
s.write_audio_chunk(0, audio_chunk)
s.write_video_chunk(1, video_chunk)
.. raw:: html
<video width="490px" controls autoplay loop muted>
<source src="https://download.pytorch.org/torchaudio/tutorial-assets/torchaudio-udp-demo.mp4">
</video>
[`code <https://download.pytorch.org/torchaudio/tutorial-assets/udp.py>`__]
.. GENERATED FROM PYTHON SOURCE LINES 329-330
Tag: :obj:`torchaudio.io`
.. rst-class:: sphx-glr-timing
**Total running time of the script:** ( 0 minutes 0.000 seconds)
.. _sphx_glr_download_tutorials_streamwriter_advanced.py:
.. only:: html
.. container:: sphx-glr-footer sphx-glr-footer-example
.. container:: sphx-glr-download sphx-glr-download-python
:download:`Download Python source code: streamwriter_advanced.py <streamwriter_advanced.py>`
.. container:: sphx-glr-download sphx-glr-download-jupyter
:download:`Download Jupyter notebook: streamwriter_advanced.ipynb <streamwriter_advanced.ipynb>`
.. only:: html
.. rst-class:: sphx-glr-signature
`Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_