Shortcuts

GELU

class torch.nn.GELU(approximate='none')[source]

Applies the Gaussian Error Linear Units function:

GELU(x)=xΦ(x)\text{GELU}(x) = x * \Phi(x)

where Φ(x)\Phi(x) is the Cumulative Distribution Function for Gaussian Distribution.

When the approximate argument is ‘tanh’, Gelu is estimated with:

GELU(x)=0.5x(1+Tanh((2/π)(x+0.044715x3)))\text{GELU}(x) = 0.5 * x * (1 + \text{Tanh}(\sqrt(2 / \pi) * (x + 0.044715 * x^3)))
Parameters:

approximate (str, optional) – the gelu approximation algorithm to use: 'none' | 'tanh'. Default: 'none'

Shape:
  • Input: ()(*), where * means any number of dimensions.

  • Output: ()(*), same shape as the input.

../_images/GELU.png

Examples:

>>> m = nn.GELU()
>>> input = torch.randn(2)
>>> output = m(input)

Docs

Access comprehensive developer documentation for PyTorch

View Docs

Tutorials

Get in-depth tutorials for beginners and advanced developers

View Tutorials

Resources

Find development resources and get your questions answered

View Resources