Specialized Lab · Dedicated ViewInformation Theory & Probability

Shannon Entropy & Information Theory

Uncertainty, optimal compression, Kullback-Leibler divergence, and channel capacity

Interactive simulator of discrete and continuous probability distributions. Manipulate Bernoulli and multinomial distributions to compute entropy in bits, informational redundancy, and lossless compression bounds.

Fundamental Law
$$H(X) = -\sum_{i=1}^n P(x_i) \log_2 P(x_i)$$
Mathematical Foundation

Shannon's Information Theory

In 1948, Claude Shannon stripped information of subjective semantic meaning, redefining it as a rigorous probabilistic measure of uncertainty reduction. Contemporary Deep Learning training is, analytically, the minimization of a Shannon divergence.

Average Uncertainty

Entropía de Shannon $H(X)$

$$H(X) = - \sum_{x \in \mathcal{X}} p(x) \log_2 p(x)$$

Quantifies expected surprise or the minimum theoretical bitrate needed to transmit states of discrete random variable $X$. Maximized by the uniform distribution ($H(X) = \log_2 |\mathcal{X}|$).

Relative Entropy

Divergencia de Kullback-Leibler $D_{\text{KL}}$

$$D_{\text{KL}}(P \parallel Q) = \sum_{x \in \mathcal{X}} P(x) \log\left(\frac{P(x)}{Q(x)}\right)$$

Measures inefficiency of encoding events from true distribution $P$ using approximating codebook $Q$. By Jensen's inequality, $D_{\text{KL}}(P \parallel Q) \ge 0$ (equality holds iff $P = Q$). Asymmetric.

Loss Objective in Deep Learning

Entropía Cruzada (Cross-Entropy Loss)

$$H(P, Q) = H(P) + D_{\text{KL}}(P \parallel Q) = - \sum_{x} P(x) \log Q(x)$$

In supervised learning and autoregressive LLMs, empirical distribution $P$ is constant ($H(P)$ fixed). Hence, minimizing cross-entropy is mathematically identical to minimizing KL divergence.

Information Theoretic Channel Limit

Información Mutua & Límite de Shannon

$$I(X; Y) = H(X) - H(X | Y), \quad C = B \log_2\left(1 + \frac{S}{N}\right)$$

$I(X; Y)$ measures the reduction in uncertainty about $X$ obtained by observing $Y$. In deep learning, the Information Bottleneck Principle states hidden layers maximize $I(Z; Y)$ while compressing $I(X; Z)$.