Automatic differentiation package  torch.autograd¶
torch.autograd provides classes and functions implementing automatic
differentiation of arbitrary scalar valued functions. It requires minimal
changes to the existing code  you only need to wrap all tensors in
Variable
objects.

torch.autograd.
backward
(variables, grad_variables=None, retain_graph=None, create_graph=None, retain_variables=None)[source]¶ Computes the sum of gradients of given variables w.r.t. graph leaves.
The graph is differentiated using the chain rule. If any of
variables
are nonscalar (i.e. their data has more than one element) and require gradient, the function additionally requires specifyinggrad_variables
. It should be a sequence of matching length, that contains gradient of the differentiated function w.r.t. corresponding variables (None
is an acceptable value for all variables that don’t need gradient tensors).This function accumulates gradients in the leaves  you might need to zero them before calling it.
Parameters:  variables (sequence of Variable) – Variables of which the derivative will be computed.
 grad_variables (sequence of (Tensor, Variable or None)) – Gradients w.r.t.
each element of corresponding variables. Any tensors will be
automatically converted to Variables that are volatile unless
create_graph
isTrue
. None values can be specified for scalar Variables or ones that don’t require grad. If a None value would be acceptable for all grad_variables, then this argument is optional.  retain_graph (bool, optional) – If
False
, the graph used to compute the grad will be freed. Note that in nearly all cases setting this option toTrue
is not needed and often can be worked around in a much more efficient way. Defaults to the value ofcreate_graph
.  create_graph (bool, optional) – If
True
, graph of the derivative will be constructed, allowing to compute higher order derivative products. Defaults toFalse
, unlessgrad_variables
contains at least one nonvolatile Variable.

torch.autograd.
grad
(outputs, inputs, grad_outputs=None, retain_graph=None, create_graph=None, only_inputs=True, allow_unused=False)[source]¶ Computes and returns the sum of gradients of outputs w.r.t. the inputs.
grad_outputs
should be a sequence of length matchingoutput
containing the precomputed gradients w.r.t. each of the outputs. If an output doesn’t require_grad, then the gradient can beNone
). Gradients can be given as Tensors when one doesn’t need the graph of the derivative, or as Variables, in which case the graph will be created.If
only_inputs
isTrue
, the function will only return a list of gradients w.r.t the specified inputs. If it’sFalse
, then gradient w.r.t. all remaining leaves will still be computed, and will be accumulated into their.grad
attribute.Parameters:  outputs (sequence of Variable) – outputs of the differentiated function.
 inputs (sequence of Variable) – Inputs w.r.t. which the gradient will be
returned (and not accumulated into
.grad
).  grad_outputs (sequence of Tensor or Variable) – Gradients w.r.t. each output.
Any tensors will be automatically converted to Variables that are
volatile unless
create_graph
isTrue
. None values can be specified for scalar Variables or ones that don’t require grad. If a None value would be acceptable for all grad_variables, then this argument is optional.  retain_graph (bool, optional) – If
False
, the graph used to compute the grad will be freed. Note that in nearly all cases setting this option toTrue
is not needed and often can be worked around in a much more efficient way. Defaults to the value ofcreate_graph
.  create_graph (bool, optional) – If
True
, graph of the derivative will be constructed, allowing to compute higher order derivative products. Defaults toFalse
, unlessgrad_variables
contains at least one nonvolatile Variable.  only_inputs (bool, optional) – If
True
, gradient w.r.t. leaves that are part of the graph, but don’t appear ininputs
won’t be computed and accumulated. Defaults toTrue
.  allow_unused (bool, optional) – If
False
, specifying inputs that were not used when computing outputs (and therefore their grad is always zero) is an error. Defaults toFalse
.
Variable¶
API compatibility¶
Variable API is nearly the same as regular Tensor API (with the exception
of a couple inplace methods, that would overwrite inputs required for
gradient computation). In most cases Tensors can be safely replaced with
Variables and the code will remain to work just fine. Because of this,
we’re not documenting all the operations on variables, and you should
refer to torch.Tensor
docs for this purpose.
Inplace operations on Variables¶
Supporting inplace operations in autograd is a hard matter, and we discourage their use in most cases. Autograd’s aggressive buffer freeing and reuse makes it very efficient and there are very few occasions when inplace operations actually lower memory usage by any significant amount. Unless you’re operating under heavy memory pressure, you might never need to use them.
Inplace correctness checks¶
All Variable
s keep track of inplace operations applied to them, and
if the implementation detects that a variable was saved for backward in one of
the functions, but it was modified inplace afterwards, an error will be raised
once backward pass is started. This ensures that if you’re using inplace
functions and not seeing any errors, you can be sure that the computed
gradients are correct.

class
torch.autograd.
Variable
[source]¶ Wraps a tensor and records the operations applied to it.
Variable is a thin wrapper around a Tensor object, that also holds the gradient w.r.t. to it, and a reference to a function that created it. This reference allows retracing the whole chain of operations that created the data. If the Variable has been created by the user, its grad_fn will be
None
and we call such objects leaf Variables.Since autograd only supports scalar valued function differentiation, grad size always matches the data size. Also, grad is normally only allocated for leaf variables, and will be always zero otherwise.
Variables:  data – Wrapped tensor of any type.
 grad – Variable holding the gradient of type and location matching
the
.data
. This attribute is lazily allocated and can’t be reassigned.  requires_grad – Boolean indicating whether the Variable has been created by a subgraph containing any Variable, that requires it. See Excluding subgraphs from backward for more details. Can be changed only on leaf Variables.
 volatile – Boolean indicating that the Variable should be used in inference mode, i.e. don’t save the history. See Excluding subgraphs from backward for more details. Can be changed only on leaf Variables.
 is_leaf – Boolean indicating if the Variable is a graph leaf (i.e if it was created by the user).
 grad_fn – Gradient function graph trace.
Parameters: 
backward
(gradient=None, retain_graph=None, create_graph=None, retain_variables=None)[source]¶ Computes the gradient of current variable w.r.t. graph leaves.
The graph is differentiated using the chain rule. If the variable is nonscalar (i.e. its data has more than one element) and requires gradient, the function additionally requires specifying
gradient
. It should be a tensor of matching type and location, that contains the gradient of the differentiated function w.r.t.self
.This function accumulates gradients in the leaves  you might need to zero them before calling it.
Parameters:  gradient (Tensor, Variable or None) – Gradient w.r.t. the
variable. If it is a tensor, it will be automatically converted
to a Variable that is volatile unless
create_graph
is True. None values can be specified for scalar Variables or ones that don’t require grad. If a None value would be acceptable then this argument is optional.  retain_graph (bool, optional) – If
False
, the graph used to compute the grads will be freed. Note that in nearly all cases setting this option to True is not needed and often can be worked around in a much more efficient way. Defaults to the value ofcreate_graph
.  create_graph (bool, optional) – If
True
, graph of the derivative will be constructed, allowing to compute higher order derivative products. Defaults toFalse
, unlessgradient
is a volatile Variable.
 gradient (Tensor, Variable or None) – Gradient w.r.t. the
variable. If it is a tensor, it will be automatically converted
to a Variable that is volatile unless

detach
()[source]¶ Returns a new Variable, detached from the current graph.
Result will never require gradient. If the input is volatile, the output will be volatile too.
Note
Returned Variable uses the same data tensor, as the original one, and inplace modifications on either of them will be seen, and may trigger errors in correctness checks.

register_hook
(hook)[source]¶ Registers a backward hook.
The hook will be called every time a gradient with respect to the variable is computed. The hook should have the following signature:
hook(grad) > Variable or None
The hook should not modify its argument, but it can optionally return a new gradient which will be used in place of
grad
.This function returns a handle with a method
handle.remove()
that removes the hook from the module.Example
>>> v = Variable(torch.Tensor([0, 0, 0]), requires_grad=True) >>> h = v.register_hook(lambda grad: grad * 2) # double the gradient >>> v.backward(torch.Tensor([1, 1, 1])) >>> v.grad.data 2 2 2 [torch.FloatTensor of size 3] >>> h.remove() # removes the hook
Function¶

class
torch.autograd.
Function
[source]¶ Records operation history and defines formulas for differentiating ops.
Every operation performed on
Variable
s creates a new function object, that performs the computation, and records that it happened. The history is retained in the form of a DAG of functions, with edges denoting data dependencies (input < output
). Then, when backward is called, the graph is processed in the topological ordering, by callingbackward()
methods of eachFunction
object, and passing returned gradients on to nextFunction
s.Normally, the only way users interact with functions is by creating subclasses and defining new operations. This is a recommended way of extending torch.autograd.
Each function is meant to be used only once (in the forward pass).
Variables: requires_grad – Boolean indicating whether the backward()
will ever need to be called.Examples:
>>> class Exp(Function): >>> >>> @staticmethod >>> def forward(ctx, i): >>> result = i.exp() >>> ctx.save_for_backward(result) >>> return result >>> >>> @staticmethod >>> def backward(ctx, grad_output): >>> result, = ctx.saved_variables >>> return grad_output * result

static
backward
(ctx, *grad_outputs)[source]¶ Defines a formula for differentiating the operation.
This function is to be overriden by all subclasses.
It must accept a context ctx as the first argument, followed by as many outputs did
forward()
return, and it should return as many tensors, as there were inputs toforward()
. Each argument is the gradient w.r.t the given output, and each returned value should be the gradient w.r.t. the corresponding input.The context can be used to retrieve variables saved during the forward pass.

static
forward
(ctx, *args, **kwargs)[source]¶ Performs the operation.
This function is to be overriden by all subclasses.
It must accept a context ctx as the first argument, followed by any number of arguments (tensors or other types).
The context can be used to store variables that can be then retrieved during the backward pass.

static
Profiler¶
Autograd includes a profiler that lets you inspect the cost of different
operators inside your model  both on the CPU and GPU. There are two modes
implemented at the moment  CPUonly using profile
.
and nvprof based (registers both CPU and GPU activity) using
emit_nvtx
.

class
torch.autograd.profiler.
profile
(enabled=True)[source]¶ Context manager that manages autograd profiler state and holds a summary of results.
Parameters: enabled (bool, optional) – Setting this to False makes this context manager a noop. Default: True
.Example
>>> x = Variable(torch.randn(1, 1), requires_grad=True) >>> with torch.autograd.profiler.profile() as prof: ... y = x ** 2 ... y.backward() >>> # NOTE: some columns were removed for brevity ... print(prof)    Name CPU time CUDA time    PowConstant 142.036us 0.000us N5torch8autograd9GraphRootE 63.524us 0.000us PowConstantBackward 184.228us 0.000us MulConstant 50.288us 0.000us PowConstant 28.439us 0.000us Mul 20.154us 0.000us N5torch8autograd14AccumulateGradE 13.790us 0.000us N5torch8autograd5CloneE 4.088us 0.000us

export_chrome_trace
(path)[source]¶ Exports an EventList as a Chrome tracing tools file.
The checkpoint can be later loaded and inspected under
chrome://tracing
URL.Parameters: path (str) – Path where the trace will be written.

key_averages
()[source]¶ Averages all function events over their keys.
Returns: An EventList containing FunctionEventAvg objects.

table
(sort_by=None)[source]¶ Prints an EventList as a nicely formatted table.
Parameters: sort_by (str, optional) – Attribute used to sort entries. By default they are printed in the same order as they were registered. Valid keys include: cpu_time
,cuda_time
,cpu_time_total
,cuda_time_total
,count
.Returns: A string containing the table.


class
torch.autograd.profiler.
emit_nvtx
(enabled=True)[source]¶ Context manager that makes every autograd operation emit an NVTX range.
It is useful when running the program under nvprof:
nvprof profilefromstart off o trace_name.prof  <regular command here>
Unfortunately, there’s no way to force nvprof to flush the data it collected to disk, so for CUDA profiling one has to use this context manager to annotate nvprof traces and wait for the process to exit before inspecting them. Then, either NVIDIA Visual Profiler (nvvp) can be used to visualize the timeline, or
torch.autograd.profiler.load_nvprof()
can load the results for inspection e.g. in Python REPL.Parameters: enabled (bool, optional) – Setting this to False makes this context manager a noop. Default: True
.Example
>>> with torch.cuda.profiler.profile(): ... model(x) # Warmup CUDA memory allocator and profiler ... with torch.autograd.profiler.emit_nvtx(): ... model(x)