Day 3: Reasoning and reinforcement learning, from MDPs to GRPO

The third day of DL4SCI 2026 was about how models learn to act and reason, rather than only to predict. The morning opened with a survey of reasoning in large language models: how to make a model "think" before it answers, how scaling laws change once inference-time compute enters the picture, and whether diffusion can replace autoregressive decoding for text. It continued with a long, carefully built tour of reinforcement learning, from the Markov decision process all the way to the recipes used to post-train today's reasoning models. The afternoon turned to vision and the physical structure hidden inside an image. Taken together, the day traced a single question from several angles: what signal should a model be optimized against, and how cheaply can we provide it?

Talks

Reasoning — Aditya Grover (UCLA)

The talk covered three connected threads: techniques that let a model spend more computation before committing to an answer, how scaling laws behave when that extra computation is counted, and diffusion as an alternative way to generate language. My notes on this session are brief, so this section only records the outline.

Reasoning techniques

The first thread was about making models "think" before answering. The techniques discussed were chain-of-thought prompting, self-consistency, search combined with verification, and, more generally, test-time compute. The common idea is that the quality of an answer need not be fixed by the forward pass alone: a model can generate intermediate steps, sample several candidate solutions and compare them, or search over possibilities and check them, trading inference cost for accuracy.

Scaling laws and inference-time compute

The second thread revisited scaling laws, the empirical relationships between loss or capability and the three classic resources: model size, data and training compute. The point emphasized was that this story shifts once inference-time compute, that is, reasoning, becomes a fourth axis. A model's effective capability then depends not only on how it was trained but also on how much computation it is allowed to spend at answer time.

Diffusion models for language

The third thread considered diffusion, or denoising, as an alternative to autoregressive decoding for language. Instead of emitting tokens strictly left to right, a diffusion language model refines a whole sequence of tokens in parallel over several iterations, starting from a noisy or masked draft and progressively cleaning it up.

Introduction to RL — Elynn Chen (NYU)

Elynn Chen walked the full arc from classical reinforcement learning to the post-training recipes behind current LLMs:

$$ \text{Q-learning} \;\rightarrow\; \text{policy optimization} \;\rightarrow\; \text{RLHF} \;\rightarrow\; \text{DPO} \;\rightarrow\; \text{RLVR} \;\rightarrow\; \text{GRPO}. $$

The through-line is that the same objective, maximize expected reward, reappears at every stage. What changes is where the reward comes from (an environment, then a learned model of human preferences, then a programmatic verifier) and how expensive the optimization is. Read in that light, each method is a response to a specific cost or limitation of the one before it.

Foundations: the Markov decision process

A reinforcement learning problem is formalized as a Markov decision process (MDP), the tuple $(\mathcal{S}, \mathcal{A}, P, r, \gamma)$. Here $\mathcal{S}$ is the set of states $s$, $\mathcal{A}$ the set of actions $a$, $P(s' \mid s, a)$ the probability of moving to state $s'$ after taking action $a$ in state $s$, $r(s,a)$ the reward received, and $\gamma \in [0,1)$ a discount factor that makes near-term rewards count more than distant ones.

The agent acts according to a policy $\pi(a \mid s)$, a distribution over actions given the current state. Writing $r_t$ for the reward at time step $t$, the return from time $t$ is the discounted sum of future rewards,

$$ G_t = \sum_{k \ge 0} \gamma^{k}\, r_{t+k}, $$

and the goal is to find the policy that maximizes the expected return from the start, $J(\pi) = \mathbb{E}_\pi[G_0]$.

Two value functions quantify how good things are under a given policy. The state value $V^\pi(s)$ is the expected return starting from $s$, and the action value $Q^\pi(s,a)$ is the expected return after taking $a$ in $s$ and following $\pi$ thereafter:

$$ V^\pi(s) = \mathbb{E}_\pi[G_t \mid s_t = s], \qquad Q^\pi(s,a) = \mathbb{E}_\pi[G_t \mid s_t = s,\, a_t = a]. $$

The whole field rests on the recursive consistency these functions satisfy, the Bellman equations. The value of an action is its immediate reward plus the discounted value of where it leads, and the optimal state value is obtained by picking the best action:

$$ Q^\pi(s,a) = r(s,a) + \gamma\, \mathbb{E}_{s'}\big[V^\pi(s')\big], \qquad V^{*}(s) = \max_a Q^{*}(s,a), $$

where $s' \sim P(\cdot \mid s, a)$ and the star denotes the optimal policy's value functions.

Value-based methods: Q-learning

The first family of methods learns $Q^{*}$ directly and then acts greedily, choosing $\arg\max_a Q(s,a)$. Q-learning does this without a model of the environment (it is model-free) and can learn from data collected by any behavior policy (it is off-policy). After observing a transition $(s, a, r, s')$ it applies a temporal-difference (TD) update with learning rate $\alpha$:

$$ Q(s,a) \;\leftarrow\; Q(s,a) + \alpha\Big[\underbrace{r + \gamma \max_{a'} Q(s',a')}_{\text{TD target}} - Q(s,a)\Big]. $$

The bracket is the TD error: the gap between a one-step bootstrapped estimate of the value and the current estimate. Replacing the table with a neural network $Q_\theta$ scales the idea to high-dimensional states; together with a replay buffer and a separate target network, this is DQN.

The limitation is the $\max_a$. It is cheap over a handful of discrete actions but awkward in large or continuous action spaces. Choosing the next token from a vocabulary of around 100k words is exactly such a case. This motivates methods that parametrize the policy directly instead of deriving it from a value function.

Policy optimization

In policy optimization the policy itself is a parametric model $\pi_\theta(a \mid s)$, and its parameters $\theta$ are updated by gradient ascent on $J(\theta)$. The policy gradient theorem gives that gradient in a form that can be estimated from sampled trajectories:

$$ \nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\big[\nabla_\theta \log \pi_\theta(a \mid s)\, A^\pi(s,a)\big], \qquad A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s). $$

The advantage $A^\pi(s,a)$ measures how much better action $a$ is than the policy's average behavior in state $s$. Intuitively, the gradient increases the log-probability of actions that turned out better than expected and decreases it for those that turned out worse.

Several algorithms differ in how they estimate and use this gradient. REINFORCE uses the raw sampled return in place of the advantage; subtracting a baseline such as $V(s)$ leaves the gradient unbiased but reduces its variance. Actor-critic methods make this structure explicit: the actor is $\pi_\theta$, and a critic network estimates $V$ or $A$.

Plain policy gradients can take destructively large steps. TRPO addresses this by taking the largest step that keeps the new policy inside a trust region, defined by a bound on the KL divergence from the old policy. PPO, the algorithm used in practice, replaces that hard constraint with a clipped surrogate objective. With the probability ratio $\rho = \pi_\theta(a \mid s) / \pi_{\theta_\text{old}}(a \mid s)$ and a small clipping parameter $\epsilon$,

$$ L^{\text{PPO}} = \mathbb{E}\big[\min\big(\rho\, A,\ \operatorname{clip}(\rho,\, 1-\epsilon,\, 1+\epsilon)\, A\big)\big]. $$

Clipping removes any incentive to move $\rho$ outside $[1-\epsilon, 1+\epsilon]$, so each update stays close to the policy that collected the data.

RLHF: reinforcement learning from human feedback

RLHF is the bridge from RL to LLM alignment, the recipe behind InstructGPT and ChatGPT. The mapping is direct: the prompt $x$ is the initial state, each generated token is an action, and producing a full response $y$ is one episode. What is missing is a reward, since there is no environment that scores a paragraph of text. RLHF supplies one in three stages.

First, supervised fine-tuning (SFT) on demonstration data produces a reasonable base policy, which serves as the reference policy $\pi_\text{ref}$. Second, humans compare pairs of responses to the same prompt, yielding preferences $y_w \succ y_l$ (the preferred "winner" over the "loser"). A reward model $r_\phi$ with parameters $\phi$ is fit to these comparisons with the Bradley-Terry loss, where $\sigma$ is the logistic sigmoid:

$$ \mathcal{L}(\phi) = -\,\mathbb{E}_{(x, y_w, y_l)}\big[\log \sigma\big(r_\phi(x,y_w) - r_\phi(x,y_l)\big)\big]. $$

Third, the policy is optimized with PPO against $r_\phi$, with a KL penalty of strength $\beta$ that keeps it close to $\pi_\text{ref}$ so it does not drift or exploit flaws in the reward model:

$$ \max_\theta\ \mathbb{E}_{x,\, y \sim \pi_\theta(\cdot \mid x)}\big[r_\phi(x,y)\big] - \beta\, \mathrm{KL}\big(\pi_\theta(\cdot \mid x)\,\|\,\pi_\text{ref}(\cdot \mid x)\big). $$

In this setup the "environment" is a learned reward model.

A distinction that is easy to blur is which of these stages is supervised and which is genuinely reinforcement learning:

The generation mechanism is the same in steps 1 and 3, next-token prediction, but the learning signal is not. In SFT the gradient says "here is the correct next token." In PPO the policy samples its own tokens with no target, and the gradient only says "this whole response scored high" or "low," as judged by the reward model.

DPO: direct preference optimization

The PPO stage of RLHF is expensive and hard to tune: it needs a reward model, online sampling from the policy, and a value network. DPO removes the RL loop entirely. Its key observation is that under the KL-regularized objective above, the optimal policy has a closed-form relationship to the reward. That relationship can be inverted to express the reward in terms of the policy itself, and substituting it into the Bradley-Terry loss turns preference learning into a classification loss directly on $\pi_\theta$:

$$ \mathcal{L}_{\text{DPO}} = -\,\mathbb{E}_{(x, y_w, y_l)}\left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_\text{ref}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_\text{ref}(y_l \mid x)}\right)\right]. $$

Each log-ratio plays the role of an implicit reward, so the loss pushes the policy to raise the relative likelihood of preferred responses over rejected ones. DPO uses the same preference data as RLHF but needs no explicit reward model and no sampling loop, which makes it more stable and cheaper.

RLVR: reinforcement learning with verifiable rewards

Both RLHF and DPO still rest on human preferences, and a learned reward model is an imperfect proxy that a policy can learn to exploit. RLVR replaces the learned reward with a programmatic verifier: is the final answer correct, does the code pass the unit tests, does the proof check? The reward becomes rule-based and binary, $+1$ for a correct output and $0$ otherwise. There is no reward model to hack, and the signal is exactly the quantity we care about.

This makes RLVR a natural fit for reasoning domains such as math and code, where correctness is checkable. It is the ingredient behind reasoning-trained models such as DeepSeek-R1-Zero, where RLVR alone, starting even from a base model, elicits long chains of thought.

GRPO: group relative policy optimization

Running PPO at LLM scale still requires a critic, a separate value network that must be trained alongside the policy at significant memory and compute cost. GRPO, introduced with DeepSeekMath and DeepSeek-R1, is a lightweight PPO variant that drops the critic. For each prompt it samples a group of $G$ responses $\{y_1, \dots, y_G\}$, scores them to obtain rewards $r_1, \dots, r_G$, and uses the group as its own baseline by normalizing each reward within the group:

$$ \hat{A}_i = \frac{r_i - \operatorname{mean}(r_1, \dots, r_G)}{\operatorname{std}(r_1, \dots, r_G)}. $$

This advantage $\hat{A}_i$ is plugged into the PPO-style clipped objective, together with a KL term toward $\pi_\text{ref}$. There is no critic to train: the baseline comes "for free" from the group.

GRPO pairs naturally with RLVR. The verifier provides a clean per-sample reward, and normalizing over a group turns those rewards into a stable advantage signal without a value model. A response that is correct while most of its siblings fail receives a large positive advantage; one that fails while most succeed receives a large negative one.

The arc at a glance

Classic RL learns values from an environment; policy optimization optimizes a policy directly; RLHF swaps the environment for a learned preference reward; DPO removes the RL loop; RLVR swaps the learned reward for a verifier; and GRPO removes the critic.

Understanding the Visual World Through Physical Intrinsics — Joseph Redmon (Ai2)

Background primer written after the school, not a record of the talk. It covers the topic in general terms and does not describe the speaker's content.

The core idea behind physical intrinsics is that an image is the end product of several physical causes, and that recovering those causes gives a representation closer to the world than raw pixels.

What are intrinsics?

A photograph is the result of several physical factors mixed together: the geometry of the scene, the materials of its surfaces, and the lighting. A camera records only their combined effect, as pixel colors. Intrinsic image decomposition, or recovering physical intrinsics, means separating those underlying factors back into per-pixel maps:

• Albedo (reflectance): the surface's own color, independent of illumination.
• Shading: how much light reaches each point, which depends on both the lighting and the geometry.
• Surface normals: the 3D orientation of the surface at each pixel.
• Depth: the distance from the camera to each point.

For diffuse (Lambertian) surfaces, the classic simplified model writes the observed intensity $I(x)$ at pixel $x$ as the product of albedo $A(x)$ and shading $S(x)$:

$$ I(x) = A(x)\, S(x). $$

Taking logarithms turns the product into a sum, $\log I = \log A + \log S$, which is why early methods treated the problem as one of signal separation. A useful analogy: a photo is a finished cake, and intrinsic decomposition tries to recover the recipe (the flour, the sugar, the oven temperature) from the cake.

Why it is hard

The decomposition is ill-posed. A dark pixel may be a dark material under bright light or a bright material in shadow, and many (albedo, shading) pairs explain the same image equally well. Solving it therefore requires priors. Classic priors assume that albedo is piecewise constant while shading varies smoothly; modern approaches learn priors from data. Real ground truth is scarce, because albedo cannot easily be measured for a real photograph, so learned methods often rely on synthetic or rendered data in which every factor is known.

Why intrinsics help visual understanding

Intrinsics sit closer to the physical world than pixels do. Albedo and geometry do not change when the lighting changes, so representations built on them are more robust to shadows, time of day and illumination. Depth and normals support 3D reasoning, describing shape and layout rather than only appearance. Intrinsics also enable editing and simulation: relighting a scene, changing a material or inserting an object consistently all require knowing them.

For science, the same idea appears as an inverse problem: inferring a hidden physical state from observations. This connects directly to the "partial observations to latent state" view of Large Physics Models discussed on Day 1.

Key takeaways

• Reasoning in LLMs spans three threads: techniques that spend more computation before answering (chain-of-thought, self-consistency, search and verification, test-time compute), scaling laws that change once inference-time compute is counted, and diffusion as an alternative to autoregressive decoding.
• The RL arc from Q-learning to GRPO keeps one objective, maximize expected reward; what changes is where the reward comes from and how expensive the optimization is.
• SFT and RL share the same next-token mechanism but not the same signal: SFT trains on the gold token, while PPO sees only a scalar reward for the whole sampled response.
• DPO removes the reward model and the sampling loop by reparametrizing the reward through the policy, turning preference learning into a classification loss.
• RLVR and GRPO together make RL on reasoning models affordable: a programmatic verifier gives a clean reward, and the group-normalized advantage removes the need for a critic.

Open questions

• How do the scaling laws from the reasoning talk change when test-time compute is added, and how does that relate to the three scaling regimes noted on Day 1?
• Where no programmatic verifier exists, as in many scientific tasks, what replaces RLVR?
• GRPO's group-normalized advantage is undefined when all $G$ responses receive the same reward, since the standard deviation is zero. How is that case handled in practice?

These are my own notes and may contain errors. Not official course material. Source on GitHub.