CUDA Stream Sanitizer¶
This is a prototype feature, which means it is at an early stage for feedback and testing, and its components are subject to change.
This module introduces CUDA Sanitizer, a tool for detecting synchronization errors between kernels ran on different streams. It stores information on accesses to tensors to determine if they are synchronized or not. When enabled in a python program and a possible data race is detected, a detailed warning will be printed and the program will exit.
It can be enabled either by importing this module and calling
enable_cuda_sanitizer() or by exporting the
Here is an example of a simple synchronization error in PyTorch:
import torch a = torch.rand(4, 2, device="cuda") with torch.cuda.stream(torch.cuda.Stream()): torch.mul(a, 5, out=a)
a tensor is initialized on the default stream and, without any synchronization
methods, modified on a new stream. The two kernels will run concurrently on the same tensor,
which might cause the second kernel to read unitialized data before the first one was able
to write it, or the first kernel might overwrite part of the result of the second.
When this script is run on the commandline with:
TORCH_CUDA_SANITIZER=1 python example_error.py
the following output is printed by CSAN:
============================ CSAN detected a possible data race on tensor with data pointer 139719969079296 Access by stream 94646435460352 during kernel: aten::mul.out(Tensor self, Tensor other, *, Tensor(a!) out) -> Tensor(a!) writing to argument(s) self, out, and to the output With stack trace: File "example_error.py", line 6, in <module> torch.mul(a, 5, out=a) ... File "pytorch/torch/cuda/_sanitizer.py", line 364, in _handle_kernel_launch stack_trace = traceback.StackSummary.extract( Previous access by stream 0 during kernel: aten::rand(int size, *, int? dtype=None, Device? device=None) -> Tensor writing to the output With stack trace: File "example_error.py", line 3, in <module> a = torch.rand(10000, device="cuda") ... File "pytorch/torch/cuda/_sanitizer.py", line 364, in _handle_kernel_launch stack_trace = traceback.StackSummary.extract( Tensor was allocated with stack trace: File "example_error.py", line 3, in <module> a = torch.rand(10000, device="cuda") ... File "pytorch/torch/cuda/_sanitizer.py", line 420, in _handle_memory_allocation traceback.StackSummary.extract(
This gives extensive insight into the origin of the error:
A tensor was incorrectly accessed from streams with ids: 0 (default stream) and 94646435460352 (new stream)
The tensor was allocated by invoking
a = torch.rand(10000, device="cuda")
- The faulty accesses were caused by operators
a = torch.rand(10000, device="cuda")on stream 0
torch.mul(a, 5, out=a)on stream 94646435460352
The error message also displays the schemas of the invoked operators, along with a note showing which arguments of the operators correspond to the affected tensor.
In the example, it can be seen that tensor
acorresponds to arguments
outputvalue of the invoked operator
The list of supported torch operators and their schemas can be viewed here.
The bug can be fixed by forcing the new stream to wait for the default stream:
with torch.cuda.stream(torch.cuda.Stream()): torch.cuda.current_stream().wait_stream(torch.cuda.default_stream()) torch.mul(a, 5, out=a)
When the script is run again, there are no errors reported.
Enables CUDA Sanitizer.
The sanitizer will begin to analyze low-level CUDA calls invoked by torch functions for synchronization errors. All data races found will be printed to the standard error output along with stack traces of suspected causes. For best results, the sanitizer should be enabled at the very beginning of the program.