Table of Contents
Shortcuts

Struct Input

Inheritance Relationships

Base Type

  • public CustomClassHolder

Struct Documentation

struct torch_tensorrt::Input : public CustomClassHolder

A struct to hold an input range (used by TensorRT Optimization profile)

This struct can either hold a single vector representing an input shape, signifying a static input shape or a set of three input shapes representing the min, optiminal and max input shapes allowed for the engine.

Public Functions

inline Input()
TORCHTRT_API Input(std::vector<int64_t> shape, TensorFormat format = TensorFormat::kContiguous)

Construct a new Input spec object for static input size from vector, optional arguments allow the user to configure expected input shape tensor format. dtype (Expected data type for the input) defaults to PyTorch / traditional TRT convection (FP32 for FP32 only, FP16 for FP32 and FP16, FP32 for Int8)

Parameters

  • shape – Input tensor shape

  • format – Expected tensor format for the input (Defaults to contiguous)

TORCHTRT_API Input(std::vector<int64_t> shape, std::vector<double> tensor_domain, TensorFormat format = TensorFormat::kContiguous)

Construct a new Input spec object for static input size from c10::ArrayRef (the type produced by tensor.sizes()), vector, optional arguments allow the user to configure expected input shape tensor format dtype (Expected data type for the input) defaults to PyTorch / traditional TRT convection (FP32 for FP32 only, FP16 for FP32 and FP16, FP32 for Int8)

Parameters

  • shape – Input tensor shape

  • tensor_domain – Allowed range for tensor inputs [low, high)

  • format – Expected tensor format for the input (Defaults to contiguous)

TORCHTRT_API Input(std::vector<int64_t> shape, DataType dtype, TensorFormat format = TensorFormat::kContiguous)

Construct a new Input spec object for static input size from vector, optional arguments allow the user to configure expected input shape tensor format.

Parameters

  • shape – Input tensor shape

  • dtype – Expected data type for the input (Defaults to the type of the weights in the first tensor calculation if detectable else Float32)

  • format – Expected tensor format for the input (Defaults to contiguous)

TORCHTRT_API Input(std::vector<int64_t> shape, DataType dtype, std::vector<double> tensor_domain, TensorFormat format = TensorFormat::kContiguous)

Construct a new Input spec object for static input size from vector, optional arguments allow the user to configure expected input shape tensor format.

Parameters

  • shape – Input tensor shape

  • dtype – Expected data type for the input (Defaults to the type of the weights in the first tensor calculation if detectable else Float32)

  • tensor_domain – Allowed range for tensor inputs [low, high)

  • format – Expected tensor format for the input (Defaults to contiguous)

TORCHTRT_API Input(c10::ArrayRef<int64_t> shape, TensorFormat format = TensorFormat::kContiguous)

Construct a new Input spec object for static input size from c10::ArrayRef (the type produced by tensor.sizes()), vector, optional arguments allow the user to configure expected input shape tensor format dtype (Expected data type for the input) defaults to PyTorch / traditional TRT convection (FP32 for FP32 only, FP16 for FP32 and FP16, FP32 for Int8)

Parameters

  • shape – Input tensor shape

  • format – Expected tensor format for the input (Defaults to contiguous)

TORCHTRT_API Input(c10::ArrayRef<int64_t> shape, std::vector<double> tensor_domain, TensorFormat format = TensorFormat::kContiguous)

Construct a new Input spec object for static input size from c10::ArrayRef (the type produced by tensor.sizes()), vector, optional arguments allow the user to configure expected input shape tensor format dtype (Expected data type for the input) defaults to PyTorch / traditional TRT convection (FP32 for FP32 only, FP16 for FP32 and FP16, FP32 for Int8)

Parameters

  • shape – Input tensor shape

  • tensor_domain – Allowed range for tensor inputs [low, high)

  • format – Expected tensor format for the input (Defaults to contiguous)

TORCHTRT_API Input(c10::ArrayRef<int64_t> shape, DataType dtype, TensorFormat format = TensorFormat::kContiguous)

Construct a new Input spec object for static input size from c10::ArrayRef (the type produced by tensor.sizes()), vector, optional arguments allow the user to configure expected input shape tensor format.

Parameters

  • shape – Input tensor shape

  • dtype – Expected data type for the input (Defaults to the type of the weights in the first tensor calculation if detectable else Float32)

  • format – Expected tensor format for the input (Defaults to contiguous)

TORCHTRT_API Input(c10::ArrayRef<int64_t> shape, DataType dtype, std::vector<double> tensor_domain, TensorFormat format = TensorFormat::kContiguous)

Construct a new Input spec object for static input size from c10::ArrayRef (the type produced by tensor.sizes()), vector, optional arguments allow the user to configure expected input shape tensor format.

Parameters

  • shape – Input tensor shape

  • dtype – Expected data type for the input (Defaults to the type of the weights in the first tensor calculation if detectable else Float32)

  • tensor_domain – Allowed range for tensor inputs [low, high)

  • format – Expected tensor format for the input (Defaults to contiguous)

TORCHTRT_API Input(std::vector<int64_t> min_shape, std::vector<int64_t> opt_shape, std::vector<int64_t> max_shape, TensorFormat format = TensorFormat::kContiguous)

Construct a new Input spec object dynamic input size from c10::ArrayRef (the type produced by tensor.sizes()) for min, opt, and max supported sizes. dtype (Expected data type for the input) defaults to PyTorch / traditional TRT convection (FP32 for FP32 only, FP16 for FP32 and FP16, FP32 for Int8)

Parameters

  • min_shape – Minimum shape for input tensor

  • opt_shape – Target optimization shape for input tensor

  • max_shape – Maximum acceptible shape for input tensor

  • format – Expected tensor format for the input (Defaults to contiguous)

TORCHTRT_API Input(std::vector<int64_t> min_shape, std::vector<int64_t> opt_shape, std::vector<int64_t> max_shape, std::vector<double> tensor_domain, TensorFormat format = TensorFormat::kContiguous)

Construct a new Input spec object dynamic input size from c10::ArrayRef (the type produced by tensor.sizes()) for min, opt, and max supported sizes. dtype (Expected data type for the input) defaults to PyTorch / traditional TRT convection (FP32 for FP32 only, FP16 for FP32 and FP16, FP32 for Int8)

Parameters

  • min_shape – Minimum shape for input tensor

  • opt_shape – Target optimization shape for input tensor

  • max_shape – Maximum acceptible shape for input tensor

  • tensor_domain – Allowed range for tensor inputs [low, high)

  • format – Expected tensor format for the input (Defaults to contiguous)

TORCHTRT_API Input(std::vector<int64_t> min_shape, std::vector<int64_t> opt_shape, std::vector<int64_t> max_shape, DataType dtype, TensorFormat format = TensorFormat::kContiguous)

Construct a new Input spec object for a dynamic input size from vectors for minimum shape, optimal shape, and max shape supported sizes optional arguments allow the user to configure expected input shape tensor format.

Parameters

  • min_shape – Minimum shape for input tensor

  • opt_shape – Target optimization shape for input tensor

  • max_shape – Maximum acceptible shape for input tensor

  • dtype – Expected data type for the input (Defaults to the type of the weights in the first tensor calculation if detectable else Float32)

  • format – Expected tensor format for the input (Defaults to contiguous)

TORCHTRT_API Input(std::vector<int64_t> min_shape, std::vector<int64_t> opt_shape, std::vector<int64_t> max_shape, DataType dtype, std::vector<double> tensor_domain, TensorFormat format = TensorFormat::kContiguous)

Construct a new Input spec object for a dynamic input size from vectors for minimum shape, optimal shape, and max shape supported sizes optional arguments allow the user to configure expected input shape tensor format.

Parameters

  • min_shape – Minimum shape for input tensor

  • opt_shape – Target optimization shape for input tensor

  • max_shape – Maximum acceptible shape for input tensor

  • dtype – Expected data type for the input (Defaults to the type of the weights in the first tensor calculation if detectable else Float32)

  • tensor_domain – Allowed range for tensor inputs [low, high)

  • format – Expected tensor format for the input (Defaults to contiguous)

TORCHTRT_API Input(c10::ArrayRef<int64_t> min_shape, c10::ArrayRef<int64_t> opt_shape, c10::ArrayRef<int64_t> max_shape, TensorFormat format = TensorFormat::kContiguous)

Construct a new Input spec object dynamic input size from c10::ArrayRef (the type produced by tensor.sizes()) for min, opt, and max supported sizes. dtype (Expected data type for the input) defaults to PyTorch / traditional TRT convection (FP32 for FP32 only, FP16 for FP32 and FP16, FP32 for Int8)

Parameters

  • min_shape – Minimum shape for input tensor

  • opt_shape – Target optimization shape for input tensor

  • max_shape – Maximum acceptible shape for input tensor

  • format – Expected tensor format for the input (Defaults to contiguous)

TORCHTRT_API Input(c10::ArrayRef<int64_t> min_shape, c10::ArrayRef<int64_t> opt_shape, c10::ArrayRef<int64_t> max_shape, std::vector<double> tensor_domain, TensorFormat format = TensorFormat::kContiguous)

Construct a new Input spec object dynamic input size from c10::ArrayRef (the type produced by tensor.sizes()) for min, opt, and max supported sizes. dtype (Expected data type for the input) defaults to PyTorch / traditional TRT convection (FP32 for FP32 only, FP16 for FP32 and FP16, FP32 for Int8)

Parameters

  • min_shape – Minimum shape for input tensor

  • opt_shape – Target optimization shape for input tensor

  • max_shape – Maximum acceptible shape for input tensor

  • tensor_domain – Allowed range for tensor inputs [low, high)

  • format – Expected tensor format for the input (Defaults to contiguous)

TORCHTRT_API Input(c10::ArrayRef<int64_t> min_shape, c10::ArrayRef<int64_t> opt_shape, c10::ArrayRef<int64_t> max_shape, DataType dtype, TensorFormat format = TensorFormat::kContiguous)

Construct a new Input spec object dynamic input size from c10::ArrayRef (the type produced by tensor.sizes()) for min, opt, and max supported sizes.

Parameters

  • min_shape – Minimum shape for input tensor

  • opt_shape – Target optimization shape for input tensor

  • max_shape – Maximum acceptible shape for input tensor

  • dtype – Expected data type for the input (Defaults to the type of the weights in the first tensor calculation if detectable else Float32)

  • format – Expected tensor format for the input (Defaults to contiguous)

TORCHTRT_API Input(c10::ArrayRef<int64_t> min_shape, c10::ArrayRef<int64_t> opt_shape, c10::ArrayRef<int64_t> max_shape, DataType dtype, std::vector<double> tensor_domain, TensorFormat format = TensorFormat::kContiguous)

Construct a new Input spec object dynamic input size from c10::ArrayRef (the type produced by tensor.sizes()) for min, opt, and max supported sizes.

Parameters

  • min_shape – Minimum shape for input tensor

  • opt_shape – Target optimization shape for input tensor

  • max_shape – Maximum acceptible shape for input tensor

  • dtype – Expected data type for the input (Defaults to the type of the weights in the first tensor calculation if detectable else Float32)

  • tensor_domain – Allowed range for tensor inputs [low, high)

  • format – Expected tensor format for the input (Defaults to contiguous)

TORCHTRT_API Input(at::Tensor tensor)

Construct a new Input spec object using a torch tensor as an example The tensor’s shape, type and layout inform the spec’s values.

Note: You cannot set dynamic shape through this method, you must use an alternative constructor

Parameters

tensor – Reference tensor to set shape, type and layout

Public Members

std::vector<int64_t> min_shape

Minimum acceptable input size into the engine.

std::vector<int64_t> opt_shape

Optimal input size into the engine (size optimized for given kernels accept any size in min max range)

std::vector<int64_t> max_shape

Maximum acceptable input size into the engine.

std::vector<int64_t> shape

Input shape to be fed to TensorRT, in the event of a dynamic shape, -1’s will hold the place of variable dimensions

DataType dtype

Expected data type for the input.

TensorFormat format

Expected tensor format for the input.

std::vector<double> tensor_domain

Expected allowed domain for tensor input.

Docs

Access comprehensive developer documentation for PyTorch

View Docs

Tutorials

Get in-depth tutorials for beginners and advanced developers

View Tutorials

Resources

Find development resources and get your questions answered

View Resources

To analyze traffic and optimize your experience, we serve cookies on this site. By clicking or navigating, you agree to allow our usage of cookies. As the current maintainers of this site, Facebook’s Cookies Policy applies. Learn more, including about available controls: Cookies Policy.