.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "tutorials/online_asr_tutorial.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_tutorials_online_asr_tutorial.py: Online ASR with Emformer RNN-T ============================== **Author**: `Jeff Hwang `__, `Moto Hira `__ This tutorial shows how to use Emformer RNN-T and streaming API to perform online speech recognition. .. GENERATED FROM PYTHON SOURCE LINES 13-28 .. note:: This tutorial requires Streaming API, FFmpeg libraries (>=4.1, <5), and SentencePiece. The Streaming API is available in nightly builds. Please refer to https://pytorch.org/get-started/locally/ for instructions. There are multiple ways to install FFmpeg libraries. If you are using Anaconda Python distribution, ``conda install 'ffmpeg<5'`` will install the required FFmpeg libraries. You can install SentencePiece by running ``pip install sentencepiece``. .. GENERATED FROM PYTHON SOURCE LINES 31-41 1. Overview ----------- Performing online speech recognition is composed of the following steps 1. Build the inference pipeline Emformer RNN-T is composed of three components: feature extractor, decoder and token processor. 2. Format the waveform into chunks of expected sizes. 3. Pass data through the pipeline. .. GENERATED FROM PYTHON SOURCE LINES 43-46 2. Preparation -------------- .. GENERATED FROM PYTHON SOURCE LINES 46-78 .. code-block:: default import IPython import torch import torchaudio try: from torchaudio.io import StreamReader except ModuleNotFoundError: try: import google.colab print( """ To enable running this notebook in Google Colab, install nightly torch and torchaudio builds and the requisite third party libraries by adding the following code block to the top of the notebook before running it: !pip3 uninstall -y torch torchvision torchaudio !pip3 install --pre torch torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cpu !pip3 install sentencepiece !add-apt-repository -y ppa:savoury1/ffmpeg4 !apt-get -qq install -y ffmpeg """ ) except ModuleNotFoundError: pass raise print(torch.__version__) print(torchaudio.__version__) .. rst-class:: sphx-glr-script-out Out: .. code-block:: none 1.12.0 0.12.0 .. GENERATED FROM PYTHON SOURCE LINES 79-88 3. Construct the pipeline ------------------------- Pre-trained model weights and related pipeline components are bundled as :py:func:`torchaudio.pipelines.RNNTBundle`. We use :py:func:`torchaudio.pipelines.EMFORMER_RNNT_BASE_LIBRISPEECH`, which is a Emformer RNN-T model trained on LibriSpeech dataset. .. GENERATED FROM PYTHON SOURCE LINES 88-95 .. code-block:: default bundle = torchaudio.pipelines.EMFORMER_RNNT_BASE_LIBRISPEECH feature_extractor = bundle.get_streaming_feature_extractor() decoder = bundle.get_decoder() token_processor = bundle.get_token_processor() .. rst-class:: sphx-glr-script-out Out: .. code-block:: none 0%| | 0.00/3.81k [00:00`__. .. GENERATED FROM PYTHON SOURCE LINES 128-135 The following audio file was originally published by LibriVox project, and it is in the public domain. https://librivox.org/great-pirate-stories-by-joseph-lewis-french/ It was re-uploaded for the sake of the tutorial. .. GENERATED FROM PYTHON SOURCE LINES 135-143 .. code-block:: default src = "https://download.pytorch.org/torchaudio/tutorial-assets/greatpiratestories_00_various.mp3" streamer = StreamReader(src) streamer.add_basic_audio_stream(frames_per_chunk=segment_length, sample_rate=bundle.sample_rate) print(streamer.get_src_stream_info(0)) print(streamer.get_out_stream_info(0)) .. rst-class:: sphx-glr-script-out Out: .. code-block:: none StreamReaderSourceAudioStream(media_type='audio', codec='mp3', codec_long_name='MP3 (MPEG audio layer 3)', format='fltp', bit_rate=128000, num_frames=0, bits_per_sample=0, sample_rate=44100.0, num_channels=2) StreamReaderOutputStream(source_index=0, filter_description='aresample=16000,aformat=sample_fmts=fltp') .. GENERATED FROM PYTHON SOURCE LINES 144-154 As previously explained, Emformer RNN-T model expects input data with overlaps; however, `Streamer` iterates the source media without overlap, so we make a helper structure that caches a part of input data from `Streamer` as right context and then appends it to the next input data from `Streamer`. The following figure illustrates this. .. image:: https://download.pytorch.org/torchaudio/tutorial-assets/emformer_rnnt_streamer_context.png .. GENERATED FROM PYTHON SOURCE LINES 154-178 .. code-block:: default class ContextCacher: """Cache the end of input data and prepend the next input data with it. Args: segment_length (int): The size of main segment. If the incoming segment is shorter, then the segment is padded. context_length (int): The size of the context, cached and appended. """ def __init__(self, segment_length: int, context_length: int): self.segment_length = segment_length self.context_length = context_length self.context = torch.zeros([context_length]) def __call__(self, chunk: torch.Tensor): if chunk.size(0) < self.segment_length: chunk = torch.nn.functional.pad(chunk, (0, self.segment_length - chunk.size(0))) chunk_with_context = torch.cat((self.context, chunk)) self.context = chunk[-self.context_length :] return chunk_with_context .. GENERATED FROM PYTHON SOURCE LINES 179-188 5. Run stream inference ----------------------- Finally, we run the recognition. First, we initialize the stream iterator, context cacher, and state and hypothesis that are used by decoder to carry over the decoding state between inference calls. .. GENERATED FROM PYTHON SOURCE LINES 188-193 .. code-block:: default cacher = ContextCacher(segment_length, context_length) state, hypothesis = None, None .. GENERATED FROM PYTHON SOURCE LINES 194-200 Next we, run the inference. For the sake of better display, we create a helper function which processes the source stream up to the given times and call it repeatedly. .. GENERATED FROM PYTHON SOURCE LINES 200-223 .. code-block:: default stream_iterator = streamer.stream() @torch.inference_mode() def run_inference(num_iter=200): global state, hypothesis chunks = [] for i, (chunk,) in enumerate(stream_iterator, start=1): segment = cacher(chunk[:, 0]) features, length = feature_extractor(segment) hypos, state = decoder.infer(features, length, 10, state=state, hypothesis=hypothesis) hypothesis = hypos[0] transcript = token_processor(hypothesis[0], lstrip=False) print(transcript, end="", flush=True) chunks.append(chunk) if i == num_iter: break return IPython.display.Audio(torch.cat(chunks).T.numpy(), rate=bundle.sample_rate) .. GENERATED FROM PYTHON SOURCE LINES 225-228 .. code-block:: default run_inference() .. rst-class:: sphx-glr-script-out Out: .. code-block:: none forward great pirate's this is aver's recording all thects recordings are in the public dum for more information or please visit liberg recording by james christopher great pirite stories by various edited by josey embodies the romance of theed expression it is a sad but inevable comment on our civilization that so far as the sea is concerned .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 230-233 .. code-block:: default run_inference() .. rst-class:: sphx-glr-script-out Out: .. code-block:: none it is developed from its infancy down to a century or so ago under one phase or another of pircy if men were savages on land they were doubly so at sea and all the years oftime adventure years that added to theap world there was little left to discover could not wholly eradate theat germ it went out gradually with the settlement of the farmed colonies great britain foremost of sea powers must be credit both directly and indirely for theation of crime and disord on the high seas than any other .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 235-238 .. code-block:: default run_inference() .. rst-class:: sphx-glr-script-out Out: .. code-block:: none force but the conquest was not complete till the advent the sea rover into the farthest corners of his domini it is said that he survised even to day in certain spots in the chinese but he is certainly an innocuous rolle a pir of any sort would be as great a curiosity to day if he could be caught and exhibited as a fab's the fact remains and will always persist that in theore of the sea he is far away the most picturesque figure in the more genuine and gross his character the higher degree of interest as he inspire there .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 240-243 .. code-block:: default run_inference() .. rst-class:: sphx-glr-script-out Out: .. code-block:: none may be a certain perversity in this for the pirate was unquestionably a bad man at his best or worst consid his surrene's andtily the worst man that ever lived there is little to soften the dark yet glowing picture of his explox but again it must be remembered that not only does the note of distant subdu and even lend a certain enchantment to the scene but the effective contrast between our peaceful times andributes much to deepen our interest in him perhaps it is this latter added to that death waspen breast that .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 245-248 .. code-block:: default run_inference() .. rst-class:: sphx-glr-script-out Out: .. code-block:: none glows at the tale of adventure which makes them the kind of hero of romance that is today he is undeni redoubtable historical figure it is a curious fact that commerce was crad in the lapane the constant danger of the deeps in this form only made heartier marin of the merchras actually stimating and strenuing maritime enterprise bucco turned for piry thus became the high romance of the sea the great centuries ofton venture it went hand in hand with disco .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 250-253 .. code-block:: default run_inference() .. rst-class:: sphx-glr-script-out Out: .. code-block:: none very they were in fact almost insepar most of mighty mariners from the days of leafou through those of the redoubtable sir francis drakeke down to our owns answered to the roll it was a bold hearty world this avarice up to the advent our giant steam every foot of which was won by fierce conquest of one sort or another out of this pass the pir emerges are romantic even at times heroic figure this final niche despite his crimes cannot altogether be denied a heroi is and all remained .. raw:: html


.. GENERATED FROM PYTHON SOURCE LINES 255-257 .. code-block:: default run_inference() .. rst-class:: sphx-glr-script-out Out: .. code-block:: none so long as tales of the are told so have him in these pages joseth lewich and of four recording by james christopher jist christ christopher yah come .. raw:: html


.. rst-class:: sphx-glr-timing **Total running time of the script:** ( 3 minutes 21.860 seconds) .. _sphx_glr_download_tutorials_online_asr_tutorial.py: .. only :: html .. container:: sphx-glr-footer :class: sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: online_asr_tutorial.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: online_asr_tutorial.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_