Day 5: Trustworthy agents for science, accuracy vs creativity in diffusion

The last day of the school paired two talks that, at first sight, live far apart: AI agents that help clinicians plan cancer treatment and assess dementia risk, and the mathematics of diffusion models. What connects them is a shared concern with when a model's output can be trusted. Sheng Liu framed trustworthiness as an engineering problem: decompose judgment into specialized agents with tools, close the loop with critique, and keep human feedback in natural language so that the reason behind a correction is not lost. Molei Tao approached the same question from the generative side, arguing that whether an output counts as a hallucination or as creativity depends on the task, and showing how the score-based view of diffusion, and its discrete masked variant, makes these properties analyzable. For anyone working in medical imaging, as I do, the day was a useful reminder that capability and reliability are separate axes.

Talks

Trustworthy AI Agents for Science — Sheng Liu (Stanford University)

The central idea of the talk is that AI agents can be useful in high-stakes scientific and clinical settings only if they are built to be verifiable: specialized experts with tools, an explicit critique loop, and a way to correct the system with expert feedback expressed in language rather than collapsed into a number. Two clinical applications, radiotherapy planning and dementia characterization, grounded the architecture, and TextGrad provided the optimization view.

Application: radiotherapy

Radiotherapy treats cancer by delivering targeted ionizing radiation to destroy tumor cells while sparing the surrounding healthy tissue and nearby organs-at-risk (OARs). The clinical pipeline offers three places where AI can assist:

• Segmentation or contouring: delineating the tumor target and the organs-at-risk on CT or MRI scans.
• Treatment planning: computing a radiation dose distribution that reaches the prescribed target dose while keeping the dose to each OAR below safe limits, which is a constrained optimization problem.
• Quality assurance: checking the plan before it is delivered.

Trustworthiness matters here in a concrete way. An over- or under-dose, or a mis-contoured organ, can harm the patient, so an agent in this loop has to be reliable, verifiable, and able to recognize when it should defer to a human.

A decision problem: treating a prostate cancer patient

The motivating clinical question is: given a specific patient, which treatment should be chosen? Prostate cancer has several valid options with different trade-offs, which makes it a genuine decision-making problem rather than a fixed protocol.

The evidence used for risk stratification includes the PSA (prostate-specific antigen) level, the Gleason score or grade group from biopsy (a measure of tumor aggressiveness), the TNM tumor stage and imaging, and patient factors such as age, comorbidities, life expectancy and preferences. The candidate actions are active surveillance (monitoring without immediate treatment, for low-risk cases), radical prostatectomy, external-beam radiotherapy and/or brachytherapy (internal radioactive seeds), and hormone therapy through androgen deprivation, often combined with radiotherapy.

This is a good test case for a trustworthy agent because the right choice balances cure probability against side effects and quality of life, the evidence is multimodal, and a wrong recommendation is costly. The agent should therefore justify its reasoning and flag its uncertainty instead of returning a single unexplained answer.

Method: a meta-optimization framing

The problem is cast as bi-level (meta-) optimization: an outer loop over meta-parameters whose objective depends on the solution of an inner loop. In general form,

$$ \min_{w}\ f_{\text{meta}}\big(w,\ \theta^{*}(w)\big) \qquad \text{subject to} \qquad \theta^{*}(w) = \arg\min_{\theta}\ g(\theta, w). $$

Here $w$ are the meta-parameters optimized by the outer loop against the meta-objective $f_{\text{meta}}$, and $\theta$ are the inner variables. For a fixed $w$, the inner loop finds the minimizer $\theta^{*}(w)$ of the inner objective $g(\theta, w)$, and that solution is fed back into $f_{\text{meta}}$. Because $f_{\text{meta}}$ is evaluated at $\theta^{*}(w)$, the outer gradient must account for how the inner solution moves when $w$ changes, which in the differentiable case means differentiating through the inner solve.

My reading of how this maps to agents: the agent itself is the object being meta-optimized. The meta-parameters $w$ define how the agent behaves (its decision policy, prompts or configuration), and the outer loop searches for the $w$ that makes the agent perform well across many tasks or patient cases rather than on one instance. Each inner solve corresponds to the agent acting on a single case under the current $w$, and $f_{\text{meta}}$ aggregates how good those per-case decisions are. In short, the outer loop learns how the agent should decide, and the inner loop is the agent deciding.

The GPT-RadPlan agents

The concrete system presented was GPT-RadPlan, "Automated radiotherapy treatment planning guided by GPT-4Vision" (Best in Medical Physics, AAPM 2024). It consists of two cooperating agents built on GPT-4Vision: an evaluator that critiques a plan and a planner that revises it.

The evaluator agent judges a radiotherapy plan through specialized expert agents, each paired with its own toolbox, whose findings are merged by an aggregation agent:

flowchart LR
    DD[dose distribution] --> VE[vision expert]
    DVH[dose-volume histogram] --> DE[DVH expert]
    VE <--> TB1[(toolbox)]
    DE <--> TB2[(toolbox)]
    VE --> AGG[aggregation agent]
    DE --> AGG
    AGG --> OUT["critique:<br/>'the dose on tumor is too low'<br/>'the DVH follows the clinical protocols'"]

The DVH expert relies on a standard summary of plan quality: the dose-volume histogram, which gives, for each dose level, the fraction or volume of a structure (target or OAR) that receives at least that dose.

The planner agent takes the current plan and the evaluator's critique and, conditioning on a memory of past evaluations, proposes new optimization parameters for the next plan:

flowchart LR
    CP[current plan] --> EV[evaluator agent]
    EV -->|evaluation| PL[planner agent]
    MEM[(memory:<br/>past evaluations)] <--> PL
    PL -->|"new parameters:<br/>w_PTV = 375, w_rectum = 2"| OPT[plan optimization]
    OPT -->|re-optimized plan| CP

The parameters are the objective weights of the treatment-planning optimization, for example $w_{\text{PTV}} = 375$ and $w_{\text{rectum}} = 2$. The PTV (planning target volume) is the tumor target plus a margin, and a high weight pushes the plan toward the prescribed tumor dose. The rectum is an organ-at-risk in prostate radiotherapy, and its weight controls how strongly the plan spares it. These weights encode the trade-off between target coverage and OAR sparing, and they are what the meta-optimization tunes.

The components fit together as follows:

The full loop is simple to state: the evaluator critiques the plan, the planner proposes new weights (for instance, raising $w_{\text{PTV}}$ after "the dose on tumor is too low"), the plan is re-optimized, and the evaluator critiques again.

Application: Cerebra for dementia

The second application was Cerebra, "A Multidisciplinary AI Board for Multimodal Dementia Characterization and Risk Assessment" (cerebra-health.com). The system reasons over the clinical spectrum

$$ \text{normal aging} \;\longrightarrow\; \text{mild cognitive impairment (MCI)} \;\longrightarrow\; \text{dementia}. $$

The motivation is stark. There are no effective drugs to treat dementia, and it is irreversible once developed. The value therefore lies in early characterization and risk assessment: identifying who is on this trajectory, for example at the MCI stage, before irreversible dementia sets in.

Cerebra is positioned as an AI co-pilot for dementia. The "multidisciplinary AI board" is a board of expert agents from different disciplines reasoning over multimodal patient data, the same pattern as the radiotherapy evaluator, where several specialists are combined into one assessment. Its stated capabilities are:

• Multi-site real-world data: more than 3M records across sites.
• Automatic model training: the system trains models on its own.
• Toolbox access: the agents can call PyTorch (deep learning), MedSAM2 (Segment Anything adapted to medical image segmentation), scikit-learn (classical ML) and FreeSurfer (brain MRI processing, including cortical reconstruction and volumetric segmentation).

Why trust needs more than capability: hallucination

Hallucination is a model producing confident but false or fabricated output. The core observation in the talk was that AI models can be intelligent and surprisingly stupid at the same time: strong on hard cases yet failing on simple ones. Capability is uneven rather than monotonic. Because those failures are hard to predict, an agent in a high-stakes clinical setting cannot be trusted on capability alone. It needs verification, tool use and explicit uncertainty so that its simple failures are caught before they are acted on.

Correcting AI with language instead of numbers

The standard way to correct a model compresses human judgment into a scalar:

$$ \text{human feedback} \;\longrightarrow\; \text{numerical label} \;\longrightarrow\; \text{model optimization}. $$

A person judges the output, the judgment becomes a score or preference, and the model is updated to increase that number, as in the RLHF loop covered on Day 3. The limitation is that collapsing rich judgment into one scalar discards the reasoning, that is, why the output was wrong.

The question posed was whether an agentic system can instead be corrected with human language:

$$ \text{expert feedback} \;\longrightarrow\; \text{natural-language feedback} \;\longrightarrow\; \text{AI agentic system}. $$

An expert critiques the system in plain language, and the agentic system consumes that critique directly to correct itself, with no scalar reward in between. The "why" is preserved.

Text gradients and TextGrad

To make this an optimization procedure, the talk drew an analogy with neural network training. Backpropagation rests on gradient descent: each variable is updated in the direction indicated by its gradient. The text-gradient version replaces the numerical gradient with a "text gradient", natural-language feedback on how to improve a variable, and lets an LLM play the role of the optimizer. Given the current variable and its feedback, the LLM produces the new variable:

flowchart LR
    V["current variable<br/>(prompts, text inputs, text outputs, ...)"] --> LLM[AI / LLM as optimizer]
    FB["text gradient<br/>= natural-language feedback"] --> LLM
    LLM --> V2[new variable]

The correspondence is direct:

Feedback also propagates backward through the computation graph of variables, mirroring the chain rule. To obtain the text gradient of a variable, the system takes the feedback already computed for the variables downstream of it in the graph (its "ancestors" in the backward pass) and generates feedback for the current variable. In backprop, a variable's gradient is assembled from the gradients of the variables it feeds into by multiplying Jacobians; here an LLM performs that composition over natural-language feedback.

This idea is implemented in TextGrad, a released Python package from the Zou group at Stanford (github.com/zou-group/textgrad). It turns natural-language feedback into an optimization signal, so an agentic system can be corrected the way a network is trained, except that the gradient is text.

From the Q&A

An audience member asked whether Cerebra, the dementia co-pilot, uses external data. The speaker's answer was no: external data is too noisy, because different doctors record different information, which makes it unreliable as an input.

My own interpretation, not stated by the speaker, is that the noise in question is inter-clinician (inter-rater) variability. For the same patient and the same clinical finding, different physicians document it differently: terminology, subjective judgment, and fields one fills in while another leaves blank. The issue is not that patients differ but that the same clinical reality becomes an unreliable input depending on who recorded it. This fits Cerebra's design, since it ingests multi-site real-world data from many clinicians, where inter-clinician noise is the central risk, and metrics computed by the agents themselves are more reproducible than human-entered external fields.

(Multi-)Modality, Accuracy and Creativity of Diffusion Models — Molei Tao (Georgia Tech)

The talk's central idea is that the score-based view of diffusion provides a mathematically clean handle on three properties of generative models: how they handle multiple modalities (including discrete data), how accurate they are, and how creative they are. A recurring point was that the line between accuracy and creativity, and therefore what counts as hallucination, is set by the task.

What generative modeling is

Generative modeling starts from samples of an unknown probability distribution, possibly in very high dimension, and aims to generate more samples that are similar, drawn from the same distribution. The speaker emphasized the word "similar" because it carries the subtlety. New samples should come from the same distribution but should not be copies of training samples. "Similar" therefore means statistically from the same distribution yet novel, and this is precisely where generalization and creativity live: too similar is memorization, too different is off-distribution.

Denoising diffusion: forward and backward processes

A denoising diffusion generative model has two processes. The forward (noising) process progressively corrupts data into noise. It is used to learn the score and is tied to the evolution of the data density $p_t(x)$ as noise is added, where $t$ indexes the noise level. The score is

$$ \nabla_x \log p_t(x), $$

the gradient of the log-density with respect to the data $x$, which points toward regions of higher density. The backward (denoising) process uses the learned score to reverse the corruption: starting from pure noise, it denoises step by step until it reaches a sample.

Score matching

The goal is to train a network $s_\theta(x, t)$, with parameters $\theta$, to approximate the true score $\nabla_x \log p_t(x)$. The naive objective, explicit score matching, is

$$ \mathbb{E}_{x \sim p}\Big[\, \big\lVert s_\theta(x) - \nabla_x \log p(x) \big\rVert^2 \,\Big], $$

which is intractable because the true score $\nabla_x \log p(x)$ is exactly what we do not know.

Denoising score matching avoids this by matching the score of the noising kernel $p(x_t \mid x_0)$, which is known, where $x_0$ is a clean data point and $x_t$ its noised version. For Gaussian noise $x_t = x_0 + \sigma \epsilon$, with noise scale $\sigma$ and $\epsilon \sim \mathcal{N}(0, I)$,

$$ \nabla_{x_t} \log p(x_t \mid x_0) = -\frac{x_t - x_0}{\sigma^2} = -\frac{\epsilon}{\sigma}. $$

Learning the score is therefore equivalent to predicting the added noise $\epsilon$, which is a tractable objective requiring no access to the true density. Minimizing denoising score matching across all noise levels yields $s_\theta \approx \nabla_x \log p_t(x)$, which the backward process then uses for generation.

Context: DDPM and score-based diffusion are the same model

The talk adopted the score-based view. As background for readers who know diffusion through DDPM, the two standard formulations compare as follows:

They are two descriptions of one model, since noise prediction and score agree up to a scale:

$$ s_\theta(x_t, t) = \nabla_{x_t} \log p_t(x_t) = -\frac{\epsilon_\theta(x_t, t)}{\sigma_t}, $$

where $\sigma_t$ is the noise scale at time $t$. DDPM's $\epsilon$-prediction is denoising score matching, up to a per-noise-level weighting. The score view is preferable for analysis for several reasons: it is more general, since the continuous SDE formulation subsumes DDPM and lets noise schedules, forward processes and samplers be chosen freely; the same score supports both a stochastic reverse SDE and a deterministic probability-flow ODE, which is fast and allows exact likelihoods; it is mathematically clean, because the score is exactly the object governing the evolution of $p_t(x)$, so accuracy, convergence and extensions can be studied rigorously; and it remains learnable thanks to denoising score matching.

Discrete diffusion versus autoregressive models

For discrete data such as text tokens, the talk contrasted an autoregressive model (GPT-2) with a masked diffusion model:

An autoregressive model factorizes a sequence $x = (x_1, \dots, x_n)$ as

$$ p(x) = \prod_{i=1}^{n} p(x_i \mid x_{<i}), $$

where $x_{<i}$ denotes the tokens before position $i$, so each step is next-token prediction in a fixed order. A masked diffusion model masks tokens in the forward process and unmasks them in the reverse. This is equivalent to any-order autoregressive generation: it does not commit to a left-to-right order, so tokens can be filled in any order and in parallel. Masked diffusion thus generalizes autoregressive modeling by relaxing the ordering constraint, and autoregression is recovered as the special case of a fixed order.

When is diffusion better than an autoregressive LM?

The answer given was not only fast generation through parallel, any-order decoding. Masked diffusion is "born for edit / repair / debug". Because it can revise tokens in any order, it naturally edits and repairs existing content in place, whereas a strict left-to-right model only appends and cannot easily go back to fix earlier tokens. The talk showed a comparison between an autoregressive and a masked-diffusion model on a task labeled VGB (AR-VGB versus MDM-VGB), with the masked diffusion model favored in the edit and repair setting.

Hallucination is task-dependent

The same generated feature can be an error in one setting and useful creativity in another. The example was a hand with six fingers: a defect if the goal is photorealism, but potentially useful for science fiction or fantasy imagery. Accuracy and creativity are two ends of the same axis, and what should be penalized as hallucination versus rewarded as creativity is defined by the task, not by the model alone. This ties back to the definition of generative modeling: the target is samples that are similar, which leaves room for novelty, and the task decides how much novelty is acceptable.

Key takeaways

• In high-stakes clinical settings such as radiotherapy, prostate cancer treatment choice and dementia risk assessment, an agent must be reliable, verifiable and able to defer, because model capability is uneven and failures on simple cases are hard to predict.
• GPT-RadPlan decomposes plan evaluation into vision and DVH experts with toolboxes plus an aggregation agent, and pairs it with a memory-equipped planner that adjusts objective weights; the whole loop is framed as bi-level meta-optimization.
• Compressing feedback into a scalar discards the reason for a correction. TextGrad keeps feedback as language, treating it as a "text gradient" that an LLM uses as optimizer and propagates backward through a computation graph.
• Denoising score matching makes the unknown score learnable by predicting known noise, and DDPM's noise prediction and score-based diffusion are the same model up to scale.
• Masked diffusion is any-order autoregression, which makes it structurally suited to editing, repairing and debugging, not only to fast parallel decoding.
• Whether an output is a hallucination or creativity depends on the task; good generation means samples from the same distribution that are not copies.

Open questions

• How exactly are $w$, $\theta$, $f_{\text{meta}}$ and $g$ instantiated in the radiotherapy setting, and is the outer loop optimized with gradients or by the LLM agents themselves?
• If natural-language feedback replaces scalar rewards, how is convergence or improvement measured, and how robust is text-gradient propagation to inconsistent expert feedback?
• Cerebra avoids external data because of inter-clinician noise; could explicit modeling of rater variability make such data usable rather than excluded?
• Can the task-dependence of hallucination be made explicit, for example by specifying how far from the training distribution a generative model is allowed to go for a given application?

These are my own notes and may contain errors. Not official course material. Source on GitHub.