Activation Functions

Popular ones

Type	Name	Formula	Common usage
Non-gated	ReLU	$\mathrm{ReLU}(x) = \max(0, x)$	Older Transformer FFNs
Non-gated	GELU	$\mathrm{GELU}(x) = x\,\Phi(x)$	BERT, GPT-2, GPT-3-style models
Non-gated	SiLU / Swish	$\mathrm{SiLU}(x) = x\,\sigma(x)$	Used directly in some modern networks
Non-gated	Squared ReLU	$\mathrm{ReLU}^2(x) = \max(0, x)^2$	Used in some newer architectures
Gated	GLU	$\sigma(xW_g) \odot (xW_u)$	Original gated linear unit
Gated	ReGLU	$\mathrm{ReLU}(xW_g) \odot (xW_u)$	GLU variant
Gated	GEGLU	$\mathrm{GELU}(xW_g) \odot (xW_u)$	T5-family / some encoder-decoder models
Gated	SwiGLU	$\mathrm{SiLU}(xW_g) \odot (xW_u)$	Very common in modern LLMs: Llama, Mistral, Qwen, etc.

Non-gated FFN

$\mathrm{FFN}(x) = f(xW_1)W_2$

Example with GELU:

$\mathrm{FFN}(x) = \mathrm{GELU}(xW_1)W_2$

Gated FFN

$\mathrm{FFN}(x) = \left[f(xW_g) \odot (xW_u)\right] W_d$

For SwiGLU specifically:

$\mathrm{FFN}(x) = \left[\mathrm{SiLU}(xW_g) \odot (xW_u)\right] W_d$

A useful modern-LLM summary is:

$\text{Older GPT/BERT} \rightarrow \text{GELU}$

$\text{Modern decoder LLMs} \rightarrow \text{SwiGLU}$

And the key naming pattern is:

$\mathrm{ReLU} \rightarrow \mathrm{ReGLU}$

$\mathrm{GELU} \rightarrow \mathrm{GEGLU}$

$\mathrm{SiLU/Swish} \rightarrow \mathrm{SwiGLU}$

---

Not so popular ones

Name	Formula	Common usage
Sigmoid	$\sigma(x) = \dfrac{1}{1 + e^{-x}}$	Gates, not usually standalone FFN activation
Tanh	$\tanh(x) = \dfrac{e^x - e^{-x}}{e^x + e^{-x}}$	Rare in modern transformer FFNs
Leaky ReLU	$\max(x, \alpha x)$	Rare in LLMs
Squared ReLU	$\mathrm{ReLU}^2(x) = \max(0, x)^2$	Some newer architectures
Mish	$x \tanh(\ln(1 + e^x))$	Rare

(source: colab)

---

Details of each activation

GELU

The exact GELU definition is:

$\mathrm{GELU}(x) = x\,\Phi(x)$

where $\Phi(x)$ is the standard normal CDF:

$\Phi(x) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{x} e^{-t^2/2}\, dt$

So equivalently:

$\mathrm{GELU}(x) = \frac{x}{2}\left(1 + \mathrm{erf}\!\left(\frac{x}{\sqrt{2}}\right)\right)$

because

$\Phi(x) = \frac{1}{2}\left(1 + \mathrm{erf}\!\left(\frac{x}{\sqrt{2}}\right)\right)$

The commonly used approximation is:

$\mathrm{GELU}(x) \approx \frac{x}{2}\left[1 + \tanh\!\left(\sqrt{\frac{2}{\pi}}\,\left(x + 0.044715\,x^3\right)\right)\right]$

Intuition:

$\mathrm{GELU}(x) = x \times \text{“probability that a standard Gaussian is} \le x\text{”}$

$\Phi(x) = P(Z \le x),\ Z \sim N(0, 1)$ — as $x \to -\infty$, $\Phi(x) \to 0$; as $x \to +\infty$, $\Phi(x) \to 1$; and $\Phi(0) = 0.5$.

SiLU

SiLU stands for Sigmoid Linear Unit.

Its mathematical definition is:

$\mathrm{SiLU}(x) = x\,\sigma(x)$

where sigmoid is

$\sigma(x) = \frac{1}{1 + e^{-x}}$

So equivalently:

$\mathrm{SiLU}(x) = \frac{x}{1 + e^{-x}}$

Gated Activation

Typically inside the Transformer FFN/MLP sublayer, not in the attention operation.

For a modern decoder block, the rough structure is:

$x \rightarrow \text{Attention} \rightarrow \text{FFN / MLP}$

Inside the FFN, a gated activation like SwiGLU is applied here:

$x \xrightarrow{\;W_g,\,W_u\;} \begin{cases} xW_g \xrightarrow{\ \mathrm{SiLU}\ } \\ xW_u \end{cases} \xrightarrow{\ \odot\ } h \xrightarrow{\;W_d\;} \text{output}$

So for SwiGLU:

$\mathrm{FFN}(x) = \left[\mathrm{SiLU}(xW_g) \odot (xW_u)\right] W_d$

The gated activation is specifically this part:

$\mathrm{SiLU}(xW_g) \odot (xW_u)$

It sits between the input/up projections and the down projection.

In code:

gate = gate_proj(x)
up   = up_proj(x)

hidden = F.silu(gate) * up   # gated activation happens here

out = down_proj(hidden)