torch.use_deterministic_algorithms

torch.use_deterministic_algorithms(mode)

Sets whether PyTorch operations must use “deterministic” algorithms. That is, algorithms which, given the same input, and when run on the same software and hardware, always produce the same output. When enabled, operations will use deterministic algorithms when available, and if only nondeterministic algorithms are available they will throw a RuntimeError when called.

The following normally-nondeterministic operations will act deterministically when mode=True:

• torch.nn.Conv1d when called on CUDA tensor

• torch.nn.Conv2d when called on CUDA tensor

• torch.nn.Conv3d when called on CUDA tensor

• torch.nn.ConvTranspose1d when called on CUDA tensor

• torch.nn.ConvTranspose2d when called on CUDA tensor

• torch.nn.ConvTranspose3d when called on CUDA tensor

• torch.bmm() when called on sparse-dense CUDA tensors

• torch.Tensor.__getitem__() when attempting to differentiate a CPU tensor and the index is a list of tensors

• torch.Tensor.index_put() with accumulate=False

• torch.Tensor.index_put() with accumulate=True when called on a CPU tensor

• torch.Tensor.put_() with accumulate=True when called on a CPU tensor

• torch.Tensor.scatter_add_() when input dimension is one and called on a CUDA tensor

• torch.gather() when input dimension is one and called on a CUDA tensor that requires grad

• torch.index_add() when called on CUDA tensor

• torch.index_select() when attempting to differentiate a CUDA tensor

• torch.repeat_interleave() when attempting to differentiate a CUDA tensor

• torch.Tensor.index_copy() when called on a CPU or CUDA tensor

The following normally-nondeterministic operations will throw a RuntimeError when mode=True:

• torch.nn.AvgPool3d when attempting to differentiate a CUDA tensor

• torch.nn.AdaptiveAvgPool2d when attempting to differentiate a CUDA tensor

• torch.nn.AdaptiveAvgPool3d when attempting to differentiate a CUDA tensor

• torch.nn.MaxPool3d when attempting to differentiate a CUDA tensor

• torch.nn.AdaptiveMaxPool2d when attempting to differentiate a CUDA tensor

• torch.nn.FractionalMaxPool2d when attempting to differentiate a CUDA tensor

• torch.nn.FractionalMaxPool3d when attempting to differentiate a CUDA tensor

• torch.nn.functional.interpolate() when attempting to differentiate a CUDA tensor and one of the following modes is used:

  • linear

  • bilinear

  • bicubic

  • trilinear

• torch.nn.ReflectionPad1d when attempting to differentiate a CUDA tensor

• torch.nn.ReflectionPad2d when attempting to differentiate a CUDA tensor

• torch.nn.ReflectionPad3d when attempting to differentiate a CUDA tensor

• torch.nn.ReplicationPad1d when attempting to differentiate a CUDA tensor

• torch.nn.ReplicationPad2d when attempting to differentiate a CUDA tensor

• torch.nn.ReplicationPad3d when attempting to differentiate a CUDA tensor

• torch.nn.NLLLoss when called on a CUDA tensor

• torch.nn.CTCLoss when attempting to differentiate a CUDA tensor

• torch.nn.EmbeddingBag when attempting to differentiate a CUDA tensor when mode='max'

• torch.Tensor.scatter_add_() when input dimension is larger than one and called on a CUDA tensor

• torch.gather() when input dimension is larger than one and called on a CUDA tensor that requires grad

• torch.Tensor.put_() when accumulate=False

• torch.Tensor.put_() when accumulate=True and called on a CUDA tensor

• torch.histc() when called on a CUDA tensor

• torch.bincount() when called on a CUDA tensor

• torch.kthvalue() with called on a CUDA tensor

• torch.median() with indices output when called on a CUDA tensor

• torch.nn.functional.grid_sample() when attempting to differentiate a CUDA tensor

A handful of CUDA operations are nondeterministic if the CUDA version is 10.2 or greater, unless the environment variable CUBLAS_WORKSPACE_CONFIG=:4096:8 or CUBLAS_WORKSPACE_CONFIG=:16:8 is set. See the CUDA documentation for more details: https://docs.nvidia.com/cuda/cublas/index.html#cublasApi_reproducibility If one of these environment variable configurations is not set, a RuntimeError will be raised from these operations when called with CUDA tensors:

• torch.mm()

• torch.mv()

• torch.bmm()

Note that deterministic operations tend to have worse performance than nondeterministic operations.

Note

This flag does not detect or prevent nondeterministic behavior caused by calling an inplace operation on a tensor with an internal memory overlap or by giving such a tensor as the out argument for an operation. In these cases, multiple writes of different data may target a single memory location, and the order of writes is not guaranteed.

Parameters

mode (bool) – If True, makes potentially nondeterministic operations switch to a deterministic algorithm or throw a runtime error. If False, allows nondeterministic operations.

Example:

>>> torch.use_deterministic_algorithms(True)

# Forward mode nondeterministic error
>>> torch.randn(10).index_copy(0, torch.tensor([0]), torch.randn(1))
...
RuntimeError: index_copy does not have a deterministic implementation...

# Backward mode nondeterministic error
>>> torch.randn(10, requires_grad=True, device='cuda').index_select(0, torch.tensor([0], device='cuda')).backward()
...
RuntimeError: index_add_cuda_ does not have a deterministic implementation...