Dynamo / torch.compile
¶
Torch-TensorRT provides a backend for the new torch.compile
API released in PyTorch 2.0. In the following examples we describe
a number of ways you can leverage this backend to accelerate inference.
Compiling ResNet using the Torch-TensorRT torch.compile Backend: Compiling a ResNet model using the Torch Compile Frontend for
torch_tensorrt.compile
Compiling a Transformer using torch.compile and TensorRT: Compiling a Transformer model using
torch.compile
Torch Compile Advanced Usage: Advanced usage including making a custom backend to use directly with the
torch.compile
APITorch Compile Stable Diffusion: Compiling a Stable Diffusion model using
torch.compile
Torch Export with Cudagraphs: Using the Cudagraphs integration with ir=”dynamo”
Using Custom Kernels within TensorRT Engines with Torch-TensorRT: Creating a plugin to use a custom kernel inside TensorRT engines
Refitting Torch-TensorRT Programs with New Weights: Refitting a compiled TensorRT Graph Module with updated weights
Mutable Torch TensorRT Module: Compile, use, and modify TensorRT Graph Module with MutableTorchTensorRTModule
vgg16_fp8_ptq: Compiling a VGG16 model with FP8 and PTQ using
torch.compile
Engine Caching: Utilizing engine caching to speed up compilation times
Engine Caching (BERT): Demonstrating engine caching on BERT
![](../../../_images/sphx_glr_refit_engine_example_thumb.png)
Refitting Torch-TensorRT Programs with New Weights
![](../../../_images/sphx_glr_torch_compile_transformers_example_thumb.png)
Compiling a Transformer using torch.compile and TensorRT
![](../../../_images/sphx_glr_torch_export_gpt2_thumb.png)
Compiling GPT2 using the Torch-TensorRT with dynamo backend
![](../../../_images/sphx_glr_torch_export_llama2_thumb.png)
Compiling Llama2 using the Torch-TensorRT with dynamo backend
![](../../../_images/sphx_glr_torch_compile_resnet_example_thumb.png)
Compiling ResNet using the Torch-TensorRT torch.compile Backend
![](../../../_images/sphx_glr_custom_kernel_plugins_thumb.png)
Using Custom Kernels within TensorRT Engines with Torch-TensorRT