# torch.onnx¶

The torch.onnx module contains functions to export models into the ONNX IR format. These models can be loaded with the ONNX library and then converted to models which run on other deep learning frameworks.

## Example: End-to-end AlexNet from PyTorch to Caffe2¶

Here is a simple script which exports a pretrained AlexNet as defined in torchvision into ONNX. It runs a single round of inference and then saves the resulting traced model to alexnet.proto:

from torch.autograd import Variable
import torch.onnx
import torchvision

dummy_input = Variable(torch.randn(10, 3, 224, 224)).cuda()
model = torchvision.models.alexnet(pretrained=True).cuda()
torch.onnx.export(model, dummy_input, "alexnet.proto", verbose=True)


The resulting alexnet.proto is a binary protobuf file which contains both the network structure and parameters of the model you exported (in this case, AlexNet). The keyword argument verbose=True causes the exporter to print out a human-readable representation of the network:

# All parameters are encoded explicitly as inputs.  By convention,
# learned parameters (ala nn.Module.state_dict) are first, and the
# actual inputs are last.
graph(%1 : Float(64, 3, 11, 11)
%2 : Float(64)
# The definition sites of all variables are annotated with type
# information, specifying the type and size of tensors.
# For example, %3 is a 192 x 64 x 5 x 5 tensor of floats.
%3 : Float(192, 64, 5, 5)
%4 : Float(192)
# ---- omitted for brevity ----
%15 : Float(1000, 4096)
%16 : Float(1000)
%17 : Float(10, 3, 224, 224)) { # the actual input!
# Every statement consists of some output tensors (and their types),
# the operator to be run (with its attributes, e.g., kernels, strides,
# etc.), its input tensors (%17, %1)
%19 : UNKNOWN_TYPE = Conv[kernels=[11, 11], strides=[4, 4], pads=[2, 2, 2, 2], dilations=[1, 1], group=1](%17, %1), uses = [[%20.i0]];
# UNKNOWN_TYPE: sometimes type information is not known.  We hope to eliminate
# all such cases in a later release.
%20 : Float(10, 64, 55, 55) = Add[broadcast=1, axis=1](%19, %2), uses = [%21.i0];
%21 : Float(10, 64, 55, 55) = Relu(%20), uses = [%22.i0];
%22 : Float(10, 64, 27, 27) = MaxPool[kernels=[3, 3], pads=[0, 0, 0, 0], dilations=[1, 1], strides=[2, 2]](%21), uses = [%23.i0];
# ...
# Finally, a network returns some tensors
return (%58);
}


You can also verify the protobuf using the onnx library. You can install onnx with conda:

conda install -c conda-forge onnx


Then, you can run:

import onnx

# Check that the IR is well formed
onnx.checker.check_model(model)

# Print a human readable representation of the graph
onnx.helper.printable_graph(model.graph)


To run the exported script with caffe2, you will need three things:

1. You’ll need an install of Caffe2. If you don’t have one already, Please follow the install instructions.

2. You’ll need onnx-caffe2, a pure-Python library which provides a Caffe2 backend for ONNX. You can install onnx-caffe2 with pip:

pip install onnx-caffe2


Once these are installed, you can use the backend for Caffe2:

# ...continuing from above
import onnx_caffe2.backend as backend
import numpy as np

rep = backend.prepare(model, device="CUDA:0") # or "CPU"
# For the Caffe2 backend:
#     rep.predict_net is the Caffe2 protobuf for the network
#     rep.workspace is the Caffe2 workspace for the network
#       (see the class onnx_caffe2.backend.Workspace)
outputs = rep.run(np.random.randn(10, 3, 224, 224).astype(np.float32))
# To run networks with more than one input, pass a tuple
# rather than a single numpy ndarray.
print(outputs[0])


In the future, there will be backends for other frameworks as well.

## Limitations¶

• The ONNX exporter is a trace-based exporter, which means that it operates by executing your model once, and exporting the operators which were actually run during this run. This means that if your model is dynamic, e.g., changes behavior depending on input data, the export won’t be accurate. Similarly, a trace is likely to be valid only for a specific input size (which is one reason why we require explicit inputs on tracing.) We recommend examining the model trace and making sure the traced operators look reasonable.
• PyTorch and Caffe2 often have implementations of operators with some numeric differences. Depending on model structure, these differences may be negligible, but they can also cause major divergences in behavior (especially on untrained models.) In a future release, we plan to allow Caffe2 to call directly to Torch implementations of operators, to help you smooth over these differences when precision is important, and to also document these differences.

## Supported operators¶

The following operators are supported:

• add (nonzero alpha not supported)
• sub (nonzero alpha not supported)
• mul
• div
• cat
• mm
• neg
• sqrt
• tanh
• sigmoid
• mean
• sum
• prod
• t
• expand (only when used before a broadcasting ONNX operator; e.g., add)
• transpose
• view
• split
• squeeze
• prelu (single weight shared among input channels not supported)
• threshold (non-zero threshold/non-zero value not supported)
• leaky_relu
• glu
• softmax
• avg_pool2d (ceil_mode not supported)
• log_softmax
• unfold (experimental support with ATen-Caffe2 integration)
• elu
• concat
• abs
• index_select
• pow
• clamp
• max
• min
• eq
• exp
• permute
• Conv
• BatchNorm
• MaxPool1d (ceil_mode not supported)
• MaxPool2d (ceil_mode not supported)
• MaxPool3d (ceil_mode not supported)
• Embedding (no optional arguments supported)
• RNN
• Dropout
• FeatureDropout (training mode not supported)
• Index (constant integer and tuple indices supported)

The operator set above is sufficient to export the following models:

• AlexNet
• DCGAN
• DenseNet
• Inception (warning: this model is highly sensitive to changes in operator implementation)
• ResNet
• SuperResolution
• VGG
• word_language_model

Adding export support for operators is an advance usage. To achieve this, developers need to touch the source code of PyTorch. Please follow the instructions for installing PyTorch from source. If the wanted operator is standardized in ONNX, it should be easy to add support for exporting such operator (adding a symbolic function for the operator). To confirm whether the operator is standardized or not, please check the ONNX operator list.

If the operator is an ATen operator, which means you can find the declaration of the function in torch/csrc/autograd/generated/VariableType.h (available in generated code in PyTorch install dir), you should add the symbolic function in torch/onnx/symbolic.py and follow the instructions listed as below:

• Define the symbolic function in torch/onnx/symbolic.py. Make sure the function has the same name as the ATen operator/function defined in VariableType.h.
• The first parameter is always the exported ONNX graph. Parameter names must EXACTLY match the names in VariableType.h, because dispatch is done with keyword arguments.
• Parameter ordering does NOT necessarily match what is in VariableType.h, tensors (inputs) are always first, then non-tensor arguments.
• In the symbolic function, if the operator is already standardized in ONNX, we only need to create a node to represent the ONNX operator in the graph.
• If the input argument is a tensor, but ONNX asks for a scalar, we have to explicitly do the conversion. The helper function _scalar can convert a scalar tensor into a python scalar, and _if_scalar_type_as can turn a Python scalar into a PyTorch tensor.

If the operator is a non-ATen operator, the symbolic function has to be added in the corresponding PyTorch Function class. Please read the following instructions:

• Create a symbolic function named symbolic in the corresponding Function class.
• The first parameter is always the exported ONNX graph.
• Parameter names except the first must EXACTLY match the names in forward.
• The output tuple size must match the outputs of forward.
• In the symbolic function, if the operator is already standardized in ONNX, we just need to create a node to represent the ONNX operator in the graph.

Symbolic functions should be implemented in Python. All of these functions interact with Python methods which are implemented via C++-Python bindings, but intuitively the interface they provide looks like this:

def operator/symbolic(g, *inputs):
"""
Modifies Graph (e.g., using "op"), adding the ONNX operations representing
this PyTorch function, and returning a Value or tuple of Values specifying the
ONNX outputs whose values correspond to the original PyTorch return values
of the autograd Function (or None if an output is not supported by ONNX).

Arguments:
g (Graph): graph to write the ONNX representation into
inputs (Value...): list of values representing the variables which contain
the inputs for this function
"""

class Value(object):
"""Represents an intermediate tensor value computed in ONNX."""
def type(self):
"""Returns the Type of the value."""

class Type(object):
def sizes(self):
"""Returns a tuple of ints representing the shape of a tensor this describes."""

class Graph(object):
def op(self, opname, *inputs, **attrs):
"""
Create an ONNX operator 'opname', taking 'args' as inputs
and attributes 'kwargs' and add it as a node to the current graph,
returning the value representing the single output of this
operator (see the outputs keyword argument for multi-return
nodes).

The set of operators and the inputs/attributes they take
is documented at https://github.com/onnx/onnx/blob/master/docs/Operators.md

Arguments:
opname (string): The ONNX operator name, e.g., Abs or Add.
args (Value...): The inputs to the operator; usually provided
as arguments to the symbolic definition.
kwargs: The attributes of the ONNX operator, with keys named
according to the following convention: alpha_f indicates
the alpha attribute with type f.  The valid type specifiers are
f (float), i (int), s (string) or t (Tensor).  An attribute
specified with type float accepts either a single float, or a
list of floats (e.g., you would say dims_i for a dims attribute
that takes a list of integers).
outputs (int, optional):  The number of outputs this operator returns;
by default an operator is assumed to return a single output.
If outputs is greater than one, this functions returns a tuple
of output Value, representing each output of the ONNX operator
in positional.
"""


The ONNX graph C++ definition is in torch/csrc/jit/ir.h.

Here is an example of handling missing symbolic function for elu operator. We try to export the model and see the error message as below:

UserWarning: ONNX export failed on elu because torch.onnx.symbolic.elu does not exist
RuntimeError: ONNX export failed: Couldn't export operator elu


The export fails because PyTorch does not support exporting elu operator. We find virtual Tensor elu(const Tensor & input, Scalar alpha, bool inplace) const override; in VariableType.h. This means elu is an ATen operator. We check the ONNX operator list, and confirm that Elu is standardized in ONNX. We add the following lines to symbolic.py:

def elu(g, input, alpha, inplace=False):
return g.op("Elu", input, alpha_f=_scalar(alpha))


Now PyTorch is able to export elu operator.

There are more examples in symbolic.py, tensor.py, padding.py.

The interface for specifying operator definitions is experimental; adventurous users should note that the APIs will probably change in a future interface.

## Functions¶

torch.onnx.export(model, args, f, export_params=True, verbose=False, training=False, input_names=None, output_names=None, aten=False)[source]

Export a model into ONNX format. This exporter runs your model once in order to get a trace of its execution to be exported; at the moment, it supports a limited set of dynamic models (e.g., RNNs.)

Parameters: model (torch.nn.Module) – the model to be exported. args (tuple of arguments) – the inputs to the model, e.g., such that model(*args) is a valid invocation of the model. Any non-Variable arguments will be hard-coded into the exported model; any Variable arguments will become inputs of the exported model, in the order they occur in args. If args is a Variable, this is equivalent to having called it with a 1-ary tuple of that Variable. (Note: passing keyword arguments to the model is not currently supported. Give us a shout if you need it.) f – a file-like object (has to implement fileno that returns a file descriptor) or a string containing a file name. A binary Protobuf will be written to this file. export_params (bool, default True) – if specified, all parameters will be exported. Set this to False if you want to export an untrained model. In this case, the exported model will first take all of its parameters as arguments, the ordering as specified by model.state_dict().values() verbose (bool, default False) – if specified, we will print out a debug description of the trace being exported. training (bool, default False) – export the model in training mode. At the moment, ONNX is oriented towards exporting models for inference only, so you will generally not need to set this to True. input_names (list of strings, default empty list) – names to assign to the input nodes of the graph, in order output_names (list of strings, default empty list) – names to assign to the output nodes of the graph, in order aten (bool, default False) – export the model in aten mode. If using aten mode, all the ops original exported by the functions in symbolic.py are exported as ATen ops.