Basic Math Used for Loss Functions

Entropy

The mathematical definition of entropy for a discrete probability distribution $p(x)$ is

$$H(p) = -\sum_{x} p(x) \log p(x)$$

Equivalently,

$$H(X) = \mathbb{E}_{x \sim p}\left[-\log p(x)\right]$$

For probabilities $p_1, \dots, p_n$:

$$H(p) = -\sum_{i=1}^{n} p_i \log p_i$$

The log base determines the unit:

• $\log_2 \rightarrow$ bits
• $\ln \rightarrow$ nats

Example: for a fair coin,

$$p = [0.5,\, 0.5]$$ $$H(p) = -\left(0.5 \log_2 0.5 + 0.5 \log_2 0.5\right) = 1 \text{ bit}$$

Intuition: entropy measures the average uncertainty / information content of a random variable. Higher entropy means the distribution is more uncertain or spread out.

Cross Entropy

$$H(p, q) = -\sum_{x} p(x) \log q(x)$$

For classification with $C$ classes:

$$H(p, q) = -\sum_{i=1}^{C} p_i \log q_i$$

KL divergence

$$D_{\mathrm{KL}}(p \| q) = \sum_{x} p(x) \log \frac{p(x)}{q(x)}$$

Equivalently,

$$D_{\mathrm{KL}}(p \| q) = \mathbb{E}_{x \sim p}\left[\log p(x) - \log q(x)\right]$$

For discrete probabilities $p_i, q_i$:

$$D_{\mathrm{KL}}(p \| q) = \sum_{i} p_i \log \frac{p_i}{q_i}$$

It connects directly to entropy and cross-entropy:

$$D_{\mathrm{KL}}(p \| q) = H(p, q) - H(p)$$

Softmax

$$\sigma : \mathbb{R}^n \rightarrow (0, 1)^n$$

defined by

$$\sigma(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^{n} e^{z_j}}, \qquad i = 1, \dots, n$$

where

$$\mathbf{z} = (z_1, \dots, z_n) \in \mathbb{R}^n .$$