torch.nn.attention.bias.causal_upper_left¶
- torch.nn.attention.bias.causal_upper_left(*size)[source]¶
Creates an upper-left triangular causal bias.
This function generates a upper-left triangular matrix to represent causal attention bias with a diagonal offset set so that the inclusive values are aligned to the upper left corner of the matrix. This equivalent to the is_causal=True argument in scaled_dot_product_attention.
The equivalent pytorch code for constructing this bias is:
torch.tril(torch.ones(size, dtype=torch.bool))
For instance, with shape=(3,4), the materialized bias tensor will be:
[[1, 0, 0, 0], [1, 1, 0, 0], [1, 1, 1, 0]]
- Parameters
size – The size of the bias matrix.
- Returns
The UPPER_LEFT triangular causal bias variant.
- Return type
