• Docs >
  • Named Tensors operator coverage
Shortcuts

Named Tensors operator coverage

Please read Named Tensors first for an introduction to named tensors.

This document is a reference for name inference, a process that defines how named tensors:

  1. use names to provide additional automatic runtime correctness checks

  2. propagate names from input tensors to output tensors

Below is a list of all operations that are supported with named tensors and their associated name inference rules.

If you don’t see an operation listed here, but it would help your use case, please search if an issue has already been filed and if not, file one.

Warning

The named tensor API is experimental and subject to change.

Supported Operations

API

Name inference rule

Tensor.abs(), torch.abs()

Keeps input names

Tensor.abs_()

Keeps input names

Tensor.acos(), torch.acos()

Keeps input names

Tensor.acos_()

Keeps input names

Tensor.add(), torch.add()

Unifies names from inputs

Tensor.add_()

Unifies names from inputs

Tensor.addmm(), torch.addmm()

Contracts away dims

Tensor.addmm_()

Contracts away dims

Tensor.addmv(), torch.addmv()

Contracts away dims

Tensor.addmv_()

Contracts away dims

Tensor.align_as()

See documentation

Tensor.align_to()

See documentation

Tensor.all(), torch.all()

None

Tensor.any(), torch.any()

None

Tensor.asin(), torch.asin()

Keeps input names

Tensor.asin_()

Keeps input names

Tensor.atan(), torch.atan()

Keeps input names

Tensor.atan2(), torch.atan2()

Unifies names from inputs

Tensor.atan2_()

Unifies names from inputs

Tensor.atan_()

Keeps input names

Tensor.bernoulli(), torch.bernoulli()

Keeps input names

Tensor.bernoulli_()

None

Tensor.bfloat16()

Keeps input names

Tensor.bitwise_not(), torch.bitwise_not()

Keeps input names

Tensor.bitwise_not_()

None

Tensor.bmm(), torch.bmm()

Contracts away dims

Tensor.bool()

Keeps input names

Tensor.byte()

Keeps input names

torch.cat()

Unifies names from inputs

Tensor.cauchy_()

None

Tensor.ceil(), torch.ceil()

Keeps input names

Tensor.ceil_()

None

Tensor.char()

Keeps input names

Tensor.chunk(), torch.chunk()

Keeps input names

Tensor.clamp(), torch.clamp()

Keeps input names

Tensor.clamp_()

None

Tensor.copy_()

out function and in-place variants

Tensor.cos(), torch.cos()

Keeps input names

Tensor.cos_()

None

Tensor.cosh(), torch.cosh()

Keeps input names

Tensor.cosh_()

None

Tensor.acosh(), torch.acosh()

Keeps input names

Tensor.acosh_()

None

Tensor.cpu()

Keeps input names

Tensor.cuda()

Keeps input names

Tensor.cumprod(), torch.cumprod()

Keeps input names

Tensor.cumsum(), torch.cumsum()

Keeps input names

Tensor.data_ptr()

None

Tensor.deg2rad(), torch.deg2rad()

Keeps input names

Tensor.deg2rad_()

None

Tensor.detach(), torch.detach()

Keeps input names

Tensor.detach_()

None

Tensor.device, torch.device()

None

Tensor.digamma(), torch.digamma()

Keeps input names

Tensor.digamma_()

None

Tensor.dim()

None

Tensor.div(), torch.div()

Unifies names from inputs

Tensor.div_()

Unifies names from inputs

Tensor.dot(), torch.dot()

None

Tensor.double()

Keeps input names

Tensor.element_size()

None

torch.empty()

Factory functions

torch.empty_like()

Factory functions

Tensor.eq(), torch.eq()

Unifies names from inputs

Tensor.erf(), torch.erf()

Keeps input names

Tensor.erf_()

None

Tensor.erfc(), torch.erfc()

Keeps input names

Tensor.erfc_()

None

Tensor.erfinv(), torch.erfinv()

Keeps input names

Tensor.erfinv_()

None

Tensor.exp(), torch.exp()

Keeps input names

Tensor.exp_()

None

Tensor.expand()

Keeps input names

Tensor.expm1(), torch.expm1()

Keeps input names

Tensor.expm1_()

None

Tensor.exponential_()

None

Tensor.fill_()

None

Tensor.flatten(), torch.flatten()

See documentation

Tensor.float()

Keeps input names

Tensor.floor(), torch.floor()

Keeps input names

Tensor.floor_()

None

Tensor.frac(), torch.frac()

Keeps input names

Tensor.frac_()

None

Tensor.ge(), torch.ge()

Unifies names from inputs

Tensor.get_device(), torch.get_device()

None

Tensor.grad

None

Tensor.gt(), torch.gt()

Unifies names from inputs

Tensor.half()

Keeps input names

Tensor.has_names()

See documentation

Tensor.index_fill(), torch.index_fill()

Keeps input names

Tensor.index_fill_()

None

Tensor.int()

Keeps input names

Tensor.is_contiguous()

None

Tensor.is_cuda

None

Tensor.is_floating_point(), torch.is_floating_point()

None

Tensor.is_leaf

None

Tensor.is_pinned()

None

Tensor.is_shared()

None

Tensor.is_signed(), torch.is_signed()

None

Tensor.is_sparse

None

Tensor.is_sparse_csr

None

torch.is_tensor()

None

Tensor.item()

None

Tensor.itemsize

None

Tensor.kthvalue(), torch.kthvalue()

Removes dimensions

Tensor.le(), torch.le()

Unifies names from inputs

Tensor.log(), torch.log()

Keeps input names

Tensor.log10(), torch.log10()

Keeps input names

Tensor.log10_()

None

Tensor.log1p(), torch.log1p()

Keeps input names

Tensor.log1p_()

None

Tensor.log2(), torch.log2()

Keeps input names

Tensor.log2_()

None

Tensor.log_()

None

Tensor.log_normal_()

None

Tensor.logical_not(), torch.logical_not()

Keeps input names

Tensor.logical_not_()

None

Tensor.logsumexp(), torch.logsumexp()

Removes dimensions

Tensor.long()

Keeps input names

Tensor.lt(), torch.lt()

Unifies names from inputs

torch.manual_seed()

None

Tensor.masked_fill(), torch.masked_fill()

Keeps input names

Tensor.masked_fill_()

None

Tensor.masked_select(), torch.masked_select()

Aligns mask up to input and then unifies_names_from_input_tensors

Tensor.matmul(), torch.matmul()

Contracts away dims

Tensor.mean(), torch.mean()

Removes dimensions

Tensor.median(), torch.median()

Removes dimensions

Tensor.nanmedian(), torch.nanmedian()

Removes dimensions

Tensor.mm(), torch.mm()

Contracts away dims

Tensor.mode(), torch.mode()

Removes dimensions

Tensor.mul(), torch.mul()

Unifies names from inputs

Tensor.mul_()

Unifies names from inputs

Tensor.mv(), torch.mv()

Contracts away dims

Tensor.names

See documentation

Tensor.narrow(), torch.narrow()

Keeps input names

Tensor.nbytes

None

Tensor.ndim

None

Tensor.ndimension()

None

Tensor.ne(), torch.ne()

Unifies names from inputs

Tensor.neg(), torch.neg()

Keeps input names

Tensor.neg_()

None

torch.normal()

Keeps input names

Tensor.normal_()

None

Tensor.numel(), torch.numel()

None

torch.ones()

Factory functions

Tensor.pow(), torch.pow()

Unifies names from inputs

Tensor.pow_()

None

Tensor.prod(), torch.prod()

Removes dimensions

Tensor.rad2deg(), torch.rad2deg()

Keeps input names

Tensor.rad2deg_()

None

torch.rand()

Factory functions

torch.rand()

Factory functions

torch.randn()

Factory functions

torch.randn()

Factory functions

Tensor.random_()

None

Tensor.reciprocal(), torch.reciprocal()

Keeps input names

Tensor.reciprocal_()

None

Tensor.refine_names()

See documentation

Tensor.register_hook()

None

Tensor.register_post_accumulate_grad_hook()

None

Tensor.rename()

See documentation

Tensor.rename_()

See documentation

Tensor.requires_grad

None

Tensor.requires_grad_()

None

Tensor.resize_()

Only allow resizes that do not change shape

Tensor.resize_as_()

Only allow resizes that do not change shape

Tensor.round(), torch.round()

Keeps input names

Tensor.round_()

None

Tensor.rsqrt(), torch.rsqrt()

Keeps input names

Tensor.rsqrt_()

None

Tensor.select(), torch.select()

Removes dimensions

Tensor.short()

Keeps input names

Tensor.sigmoid(), torch.sigmoid()

Keeps input names

Tensor.sigmoid_()

None

Tensor.sign(), torch.sign()

Keeps input names

Tensor.sign_()

None

Tensor.sgn(), torch.sgn()

Keeps input names

Tensor.sgn_()

None

Tensor.sin(), torch.sin()

Keeps input names

Tensor.sin_()

None

Tensor.sinh(), torch.sinh()

Keeps input names

Tensor.sinh_()

None

Tensor.asinh(), torch.asinh()

Keeps input names

Tensor.asinh_()

None

Tensor.size()

None

Tensor.softmax(), torch.softmax()

Keeps input names

Tensor.split(), torch.split()

Keeps input names

Tensor.sqrt(), torch.sqrt()

Keeps input names

Tensor.sqrt_()

None

Tensor.squeeze(), torch.squeeze()

Removes dimensions

Tensor.std(), torch.std()

Removes dimensions

torch.std_mean()

Removes dimensions

Tensor.stride()

None

Tensor.sub(), torch.sub()

Unifies names from inputs

Tensor.sub_()

Unifies names from inputs

Tensor.sum(), torch.sum()

Removes dimensions

Tensor.tan(), torch.tan()

Keeps input names

Tensor.tan_()

None

Tensor.tanh(), torch.tanh()

Keeps input names

Tensor.tanh_()

None

Tensor.atanh(), torch.atanh()

Keeps input names

Tensor.atanh_()

None

torch.tensor()

Factory functions

Tensor.to()

Keeps input names

Tensor.topk(), torch.topk()

Removes dimensions

Tensor.transpose(), torch.transpose()

Permutes dimensions

Tensor.trunc(), torch.trunc()

Keeps input names

Tensor.trunc_()

None

Tensor.type()

None

Tensor.type_as()

Keeps input names

Tensor.unbind(), torch.unbind()

Removes dimensions

Tensor.unflatten()

See documentation

Tensor.uniform_()

None

Tensor.var(), torch.var()

Removes dimensions

torch.var_mean()

Removes dimensions

Tensor.zero_()

None

torch.zeros()

Factory functions

Keeps input names

All pointwise unary functions follow this rule as well as some other unary functions.

  • Check names: None

  • Propagate names: input tensor’s names are propagated to the output.

>>> x = torch.randn(3, 3, names=('N', 'C'))
>>> x.abs().names
('N', 'C')

Removes dimensions

All reduction ops like sum() remove dimensions by reducing over the desired dimensions. Other operations like select() and squeeze() remove dimensions.

Wherever one can pass an integer dimension index to an operator, one can also pass a dimension name. Functions that take lists of dimension indices can also take in a list of dimension names.

  • Check names: If dim or dims is passed in as a list of names, check that those names exist in self.

  • Propagate names: If the dimensions of the input tensor specified by dim or dims are not present in the output tensor, then the corresponding names of those dimensions do not appear in output.names.

>>> x = torch.randn(1, 3, 3, 3, names=('N', 'C', 'H', 'W'))
>>> x.squeeze('N').names
('C', 'H', 'W')

>>> x = torch.randn(3, 3, 3, 3, names=('N', 'C', 'H', 'W'))
>>> x.sum(['N', 'C']).names
('H', 'W')

# Reduction ops with keepdim=True don't actually remove dimensions.
>>> x = torch.randn(3, 3, 3, 3, names=('N', 'C', 'H', 'W'))
>>> x.sum(['N', 'C'], keepdim=True).names
('N', 'C', 'H', 'W')

Unifies names from inputs

All binary arithmetic ops follow this rule. Operations that broadcast still broadcast positionally from the right to preserve compatibility with unnamed tensors. To perform explicit broadcasting by names, use Tensor.align_as().

  • Check names: All names must match positionally from the right. i.e., in tensor + other, match(tensor.names[i], other.names[i]) must be true for all i in (-min(tensor.dim(), other.dim()) + 1, -1].

  • Check names: Furthermore, all named dimensions must be aligned from the right. During matching, if we match a named dimension A with an unnamed dimension None, then A must not appear in the tensor with the unnamed dimension.

  • Propagate names: unify pairs of names from the right from both tensors to produce output names.

For example,

# tensor: Tensor[   N, None]
# other:  Tensor[None,    C]
>>> tensor = torch.randn(3, 3, names=('N', None))
>>> other = torch.randn(3, 3, names=(None, 'C'))
>>> (tensor + other).names
('N', 'C')

Check names:

  • match(tensor.names[-1], other.names[-1]) is True

  • match(tensor.names[-2], tensor.names[-2]) is True

  • Because we matched None in tensor with 'C', check to make sure 'C' doesn’t exist in tensor (it does not).

  • Check to make sure 'N' doesn’t exists in other (it does not).

Finally, the output names are computed with [unify('N', None), unify(None, 'C')] = ['N', 'C']

More examples:

# Dimensions don't match from the right:
# tensor: Tensor[N, C]
# other:  Tensor[   N]
>>> tensor = torch.randn(3, 3, names=('N', 'C'))
>>> other = torch.randn(3, names=('N',))
>>> (tensor + other).names
RuntimeError: Error when attempting to broadcast dims ['N', 'C'] and dims
['N']: dim 'C' and dim 'N' are at the same position from the right but do
not match.

# Dimensions aren't aligned when matching tensor.names[-1] and other.names[-1]:
# tensor: Tensor[N, None]
# other:  Tensor[      N]
>>> tensor = torch.randn(3, 3, names=('N', None))
>>> other = torch.randn(3, names=('N',))
>>> (tensor + other).names
RuntimeError: Misaligned dims when attempting to broadcast dims ['N'] and
dims ['N', None]: dim 'N' appears in a different position from the right
across both lists.

Note

In both of the last examples, it is possible to align the tensors by names and then perform the addition. Use Tensor.align_as() to align tensors by name or Tensor.align_to() to align tensors to a custom dimension ordering.

Permutes dimensions

Some operations, like Tensor.t(), permute the order of dimensions. Dimension names are attached to individual dimensions so they get permuted as well.

If the operator takes in positional index dim, it is also able to take a dimension name as dim.

  • Check names: If dim is passed as a name, check that it exists in the tensor.

  • Propagate names: Permute dimension names in the same way as the dimensions that are being permuted.

>>> x = torch.randn(3, 3, names=('N', 'C'))
>>> x.transpose('N', 'C').names
('C', 'N')

Contracts away dims

Matrix multiply functions follow some variant of this. Let’s go through torch.mm() first and then generalize the rule for batch matrix multiplication.

For torch.mm(tensor, other):

  • Check names: None

  • Propagate names: result names are (tensor.names[-2], other.names[-1]).

>>> x = torch.randn(3, 3, names=('N', 'D'))
>>> y = torch.randn(3, 3, names=('in', 'out'))
>>> x.mm(y).names
('N', 'out')

Inherently, a matrix multiplication performs a dot product over two dimensions, collapsing them. When two tensors are matrix-multiplied, the contracted dimensions disappear and do not show up in the output tensor.

torch.mv(), torch.dot() work in a similar way: name inference does not check input names and removes the dimensions that are involved in the dot product:

>>> x = torch.randn(3, 3, names=('N', 'D'))
>>> y = torch.randn(3, names=('something',))
>>> x.mv(y).names
('N',)

Now, let’s take a look at torch.matmul(tensor, other). Assume that tensor.dim() >= 2 and other.dim() >= 2.

  • Check names: Check that the batch dimensions of the inputs are aligned and broadcastable. See Unifies names from inputs for what it means for the inputs to be aligned.

  • Propagate names: result names are obtained by unifying the batch dimensions and removing the contracted dimensions: unify(tensor.names[:-2], other.names[:-2]) + (tensor.names[-2], other.names[-1]).

Examples:

# Batch matrix multiply of matrices Tensor['C', 'D'] and Tensor['E', 'F'].
# 'A', 'B' are batch dimensions.
>>> x = torch.randn(3, 3, 3, 3, names=('A', 'B', 'C', 'D'))
>>> y = torch.randn(3, 3, 3, names=('B', 'E', 'F'))
>>> torch.matmul(x, y).names
('A', 'B', 'C', 'F')

Finally, there are fused add versions of many matmul functions. i.e., addmm() and addmv(). These are treated as composing name inference for i.e. mm() and name inference for add().

Factory functions

Factory functions now take a new names argument that associates a name with each dimension.

>>> torch.zeros(2, 3, names=('N', 'C'))
tensor([[0., 0., 0.],
        [0., 0., 0.]], names=('N', 'C'))

out function and in-place variants

A tensor specified as an out= tensor has the following behavior:

  • If it has no named dimensions, then the names computed from the operation get propagated to it.

  • If it has any named dimensions, then the names computed from the operation must be exactly equal to the existing names. Otherwise, the operation errors.

All in-place methods modify inputs to have names equal to the computed names from name inference. For example:

>>> x = torch.randn(3, 3)
>>> y = torch.randn(3, 3, names=('N', 'C'))
>>> x.names
(None, None)

>>> x += y
>>> x.names
('N', 'C')

Docs

Access comprehensive developer documentation for PyTorch

View Docs

Tutorials

Get in-depth tutorials for beginners and advanced developers

View Tutorials

Resources

Find development resources and get your questions answered

View Resources