Specialized Lab · Dedicated ViewReinforcement Learning & Optimization

Bellman Equation & Value Iteration (GridWorld)

Sequential optimization, optimality principle, and fixed-point operator convergence

Visual laboratory for Markov Decision Processes (MDP) and dynamic programming. Adjust discount factor gamma, compute value iterations step by step with the Bellman operator, and visualize optimal policy convergence.

Fundamental Law
$$V^*(s) = \max_a \left[ \mathcal{R}(s, a) + \gamma \sum_{s'} \mathcal{P}(s'|s,a) V^*(s') \right]$$
Dynamic Optimization & Reinforcement Learning

Bellman Equation Simulator (GridWorld & Value Iteration)

$V^*(s) = \max_a [ \mathcal{R} + \gamma \sum \mathcal{P} V^* ]$

{lang === 'es' ? 'En 1957, Richard Bellman introdujo el Principio de Optimalidad: una política óptima descompone una decisión secuencial en una recompensa inmediata más el valor esperado descontado del siguiente estado. En este entorno GridWorld, ajusta el factor de descuento temporal $\\gamma$, ejecuta iteraciones del operador de Bellman paso a paso ($V_{k+1} = \\mathcal{T} V_k$) o calcula el punto fijo para observar cómo converge la política óptima $\\pi^*(s)$ hacia la meta sin caer en la trampa.' : 'In 1957, Richard Bellman introduced the Principle of Optimality: an optimal policy decomposes sequential decisions into an immediate reward plus the discounted expected return of successor states. In this GridWorld environment, adjust the discount factor $\\gamma$, step through the Bellman operator ($V_{k+1} = \\mathcal{T} V_k$), or solve for the fixed point to observe the convergence of optimal policy $\\pi^*(s)$.'}

{lang === 'es' ? 'Meta (+1.00)' : 'Goal (+1.00)'} {lang === 'es' ? 'Trampa (-1.00)' : 'Trap (-1.00)'} {lang === 'es' ? 'Muro' : 'Wall'} ↑ {lang === 'es' ? 'Política Óptima $\\pi^*(s)$' : 'Optimal Policy $\\pi^*(s)$'}
{lang === 'es' ? 'Factor de Descuento ($\\gamma$):' : 'Discount Factor ($\\gamma$):'} 0.90
0.10 (Myopic / Immediate)0.99 (Strategic Planning)
Iteration ($k$)0
{lang === 'es' ? 'Residuo $\\|V_{k+1} - V_k\\|_\\infty$' : 'Residual $\\|V_{k+1} - V_k\\|_\\infty$'}0.0000
Banach StateWaiting to start

Analytical Foundations: From Principle of Optimality to Banach Spaces

The Bellman equation is not an ad-hoc heuristic; it is a contractive fixed-point equation with unique existence guaranteed over a complete Banach metric space.

Principle of Optimality (1957)

{lang === 'es' ? 'Ecuación de Bellman para $V^*(s)$ y $Q^*(s, a)$' : 'Bellman Equation for $V^*(s)$ & $Q^*(s, a)$'}

{lang === 'es' ? 'En un Proceso de Decisión de Markov (MDP) definido por $(\\mathcal{S}, \\mathcal{A}, \\mathcal{P}, \\mathcal{R}, \\gamma)$, el valor óptimo de estado $V^*(s)$ y la función acción-valor $Q^*(s, a)$ se definen como:' : 'In a Markov Decision Process (MDP) specified by tuple $(\\mathcal{S}, \\mathcal{A}, \\mathcal{P}, \\mathcal{R}, \\gamma)$, the optimal state value $V^*(s)$ and state-action value $Q^*(s, a)$ satisfy:'}

$$V^*(s) = \max_{a \in \mathcal{A}} \left[ \mathcal{R}(s, a) + \gamma \sum_{s' \in \mathcal{S}} \mathcal{P}(s' \mid s, a) V^*(s') \right]$$ $$Q^*(s, a) = \mathcal{R}(s, a) + \gamma \max_{a' \in \mathcal{A}} Q^*(s', a')$$

{lang === 'es' ? 'La política óptima surge tomando el argumento maximizador: $\\pi^*(s) = \\arg\\max_{a} Q^*(s, a)$.' : 'The optimal deterministic policy is recovered via greedy argument maximization: $\\pi^*(s) = \\arg\\max_{a} Q^*(s, a)$.'}

Functional Analysis

The Banach Fixed-Point Theorem

{lang === 'es' ? 'Definimos el operador de Bellman $\\mathcal{T}: \\mathbb{R}^{|\\mathcal{S}|} \\to \\mathbb{R}^{|\\mathcal{S}|}$. Para cualquier par de funciones de valor $U, V \\in \\mathbb{R}^{|\\mathcal{S}|}$, se cumple bajo la norma del supremo:' : 'Define the dynamic programming operator $\\mathcal{T}: \\mathbb{R}^{|\\mathcal{S}|} \\to \\mathbb{R}^{|\\mathcal{S}|}$. For any candidate value functions $U, V \\in \\mathbb{R}^{|\\mathcal{S}|}$, the supremum norm establishes:'}

$$\|\mathcal{T} U - \mathcal{T} V\|_\infty \le \gamma \|U - V\|_\infty, \quad \text{con } \gamma \in [0, 1)$$

{lang === 'es' ? 'Al ser $\\gamma < 1$, $\\mathcal{T}$ es una $\\gamma$-contracción estricta. Por el Teorema del Punto Fijo de Banach, existe un único punto fijo $V^*$ tal que $\\mathcal{T}V^* = V^*$, asegurando convergencia geométrica independiente de $V_0$.' : 'Since discount factor $\\gamma < 1$, mapping $\\mathcal{T}$ is a strict $\\gamma$-contraction. By the Banach Fixed-Point Theorem, there exists a unique fixed point $V^*$ satisfying $\\mathcal{T}V^* = V^*$, ensuring monotonic geometric convergence from arbitrary $V_0$.'}

Genealogy Toward Frontier AI

From Tabular Dynamic Programming to RLHF in Large Language Models

1957 · Richard BellmanDynamic Programming

Analytic derivation of the principle of optimality at the RAND Corporation.

1989 · WatkinsQ-Learning

{lang === 'es' ? 'Diferencias temporales libres de modelo ($TD$) para aprender sin conocer $\\mathcal{P}$.' : 'Model-free temporal difference control ($TD$) optimizing policies without transition knowledge $\\mathcal{P}$.'}

2015 · DeepMindDeep Q-Networks (DQN)

{lang === 'es' ? 'Aproximación de $Q(s, a; \\theta)$ con redes neuronales profundas y Experience Replay.' : 'Deep parametric function approximation $Q(s, a; \\theta)$ with stabilized Experience Replay.'}

2017+ · OpenAIPPO & RLHF

Alignment of LLMs (GPT, Claude) steering generative policies via human reward models and KL penalties.