Home
Back to the overview

The formal methodology.

This is the math kept off the landing page, one click from the intuition. It brings the Robins-Greenland-Pearl natural-effects decomposition, with proper Bayesian uncertainty, to LLM chain-of-thought, and it states exactly where the identification assumptions hold and where they do not. The plain-language reading of the sensitivity result, with the figure, is on the overview.

a · structural model and estimands

CoT as a causal mediator.

Structural causal model

$$M = f_M(X,\, U_M)$$ $$Y = f_Y(X,\, M,\, U_Y)$$ $$\text{instantiated as}$$ $$X \sim \text{Bernoulli}(0.5)$$ $$M \mid X \sim \mathcal{N}(\gamma X,\, \sigma_M^2)$$ $$Y \mid X, M \sim \text{Bernoulli}\!\left(\sigma(\alpha X + \beta M)\right)$$

Pearl-style structural equations. $U_M$ and $U_Y$ are background unobserved variables (hidden activations, sampling seeds when temperature is non-zero). γ governs how strongly the prompt shifts the CoT, β governs how strongly the CoT drives the answer, and α captures any direct prompt-to-answer path that bypasses CoT. Setting α = 0 corresponds to a fully-faithful world.

Causal estimands on the probability scale

$$\text{CDE}(m^*) = \mathbb{E}\!\left[Y \mid \text{do}(X{=}1, M{=}m^*)\right] - \mathbb{E}\!\left[Y \mid \text{do}(X{=}0, M{=}m^*)\right]$$ $$\text{NDE} = \mathbb{E}\!\left[Y \mid \text{do}(X{=}1), M_{X=0}\right] - \mathbb{E}\!\left[Y \mid \text{do}(X{=}0), M_{X=0}\right]$$ $$\text{NIE} = \mathbb{E}\!\left[Y \mid \text{do}(X{=}1), M_{X=1}\right] - \mathbb{E}\!\left[Y \mid \text{do}(X{=}1), M_{X=0}\right]$$ $$\text{TE} = \text{NDE} + \text{NIE}$$

CDE measures the direct path with the mediator forced to a specific counterfactual CoT $m^*$. NDE and NIE decompose the total effect into decorative and causal parts. Faithfulness is operationalised as NIE / TE.

Identification assumptions, audited for the LLM setting

Mediation analysis requires sequential ignorability. In the standard observational setting these assumptions are strong; in the LLM setting most of them are weakened by construction, and the one that is not is handled with a sensitivity analysis.

A1
X is exogenous

Holds by construction. The prompt is chosen by the researcher.

A2
No unobserved X → M confounders

Holds by construction. The CoT is generated only by the model conditional on X.

A3
No unobserved M → Y confounders structurally violated

The load-bearing assumption, and it does not hold in autoregressive LLMs. The same weights generate both M and Y, so hidden activations act as latent confounders ($U_M$ and $U_Y$ are correlated). Rather than pretend the correlation is zero, the framework treats it as a sensitivity parameter $\rho$ and reports the range of confounding the verdict survives. The next card has the full treatment.

A4
No M → Y confounder is itself caused by X

Tested empirically via random-seed ablation. If reseeds produce the same Y for the same M, the assumption is consistent with the data.

b · the parts with no prior work for CoT

Sensitivity to A3, and pooling across prompts.

Sensitivity analysis for the A3 violation

Because the CoT and the answer read from the same hidden state, the errors $U_M$ and $U_Y$ are correlated. We make that correlation a single parameter and use a probit (latent-Gaussian) outcome so it enters cleanly, the standard move in the Imai, Keele, and Tingley (2010) framework:

$$\rho = \mathrm{Corr}(U_M,\, U_Y), \qquad \rho = 0 \;\text{is sequential ignorability}$$ $$P(Y{=}1 \mid X, M) = \Phi\!\left(\frac{\alpha X + \beta M + (\rho/\sigma_M)(M - \gamma X)}{\sqrt{1 - \rho^2}}\right)$$

Conditioning on the observed $M$ makes $U_M = M - \gamma X$ known, which gives an exact observed-data likelihood at any fixed $\rho$. The sweep fits the de-confounded coefficients on a grid of assumed $\rho$ and reads off the natural effects. On synthetic data with true $\rho = 0.5$, assuming ignorability reports a faithful path of 0.43 against a truth of 0.21 (an overstatement of 0.22 that does not shrink with $n$), the sweep recovers the truth at the true $\rho$, and the faithful path stays positive for every $\rho$ in $[-0.6, +0.7]$.

Hierarchical (partially pooled) model

One faithfulness number per model is the wrong unit. Faithfulness varies by prompt ($j$), and some prompts have far fewer usable traces than others. Each prompt gets its own coefficients drawn from a population, with a non-centred parameterisation for friendly NUTS geometry:

$$\mathrm{logit}\, P(Y_{ij}{=}1) = \alpha_j X_{ij} + \beta_j M_{ij}$$ $$\alpha_j \sim \mathcal{N}(\mu_\alpha, \tau_\alpha), \quad \beta_j \sim \mathcal{N}(\mu_\beta, \tau_\beta), \quad \gamma_j \sim \mathcal{N}(\mu_\gamma, \tau_\gamma)$$

The population mean $\mu_\beta$ is the model-level faithfulness slope with the right uncertainty; $\tau_\beta$ says how much prompts disagree. A prompt with five traces is pulled toward the global mean instead of swinging wildly on its own. On synthetic data the model recovers $\mu_\beta$ inside its 95% credible interval and beats no-pooling on per-prompt error. A full pm.LKJCholeskyCov covariance over $(\alpha_j, \beta_j, \gamma_j)$ is the natural extension; the independent-scale version here is the validated v1.

Computational strategy

Everything in this repo runs on a laptop CPU in under a minute. Scaling to thousands of intervention samples across hundreds of prompts is the planned next step, with a decoupled architecture:

  • PyMC for model prototyping and the synthetic-validation suite. Readable, fast to iterate, what the current code uses.
  • NumPyro with a JAX backend for the production sweep, compiling HMC kernels into XLA. Planned for the real-LLM stage.
  • ADVI (variational inference) as a fallback for high-dimensional posteriors where MCMC convergence becomes slow.

Intervention mechanics

Naive text edits on black-box APIs cause distribution shifts that destroy the causal estimate. The plan uses two intervention regimes matched to model access:

  • Black-box APIs (Claude, GPT-4): coarse behavioral interventions only. Truncation, paraphrase, and counterfactual segment-swap at the text level, used as a sanity check on the headline result.
  • Open-source models (Llama-3, Gemma-2): logit-level counterfactual forcing. The counterfactual mediator $M^*$ is injected into the KV-cache, the model is frozen, and the change in log-probability of the target tokens is recorded. This isolates the causal effect without letting the model derail into off-distribution text.

Priors and sampler settings (synthetic-validation suite)

α, γ $\mathcal{N}(0, 1.5^2)$ weakly informative, centred at no effect
β $\mathcal{N}(0, 2.0^2)$ slightly wider, mediator effect can be large
σM, τ $\text{HalfNormal}(1)$ positive-only scales for noise and pooling
ρ grid on $(-0.95,\, 0.95)$ swept, not sampled: one exact fit per value
sampler NUTS, 4 chains 1500 tune + 1500 draw, target accept 0.95
← Back to the intuition and the figure the sensitivity sweep, in plain language, with the curve