This is the math kept off the landing page, one click from the intuition. It brings the Robins-Greenland-Pearl natural-effects decomposition, with proper Bayesian uncertainty, to LLM chain-of-thought, and it states exactly where the identification assumptions hold and where they do not. The plain-language reading of the sensitivity result, with the figure, is on the overview.
Pearl-style structural equations. $U_M$ and $U_Y$ are background unobserved variables (hidden activations, sampling seeds when temperature is non-zero). γ governs how strongly the prompt shifts the CoT, β governs how strongly the CoT drives the answer, and α captures any direct prompt-to-answer path that bypasses CoT. Setting α = 0 corresponds to a fully-faithful world.
CDE measures the direct path with the mediator forced to a specific counterfactual CoT $m^*$. NDE and NIE decompose the total effect into decorative and causal parts. Faithfulness is operationalised as NIE / TE.
Mediation analysis requires sequential ignorability. In the standard observational setting these assumptions are strong; in the LLM setting most of them are weakened by construction, and the one that is not is handled with a sensitivity analysis.
Holds by construction. The prompt is chosen by the researcher.
Holds by construction. The CoT is generated only by the model conditional on X.
The load-bearing assumption, and it does not hold in autoregressive LLMs. The same weights generate both M and Y, so hidden activations act as latent confounders ($U_M$ and $U_Y$ are correlated). Rather than pretend the correlation is zero, the framework treats it as a sensitivity parameter $\rho$ and reports the range of confounding the verdict survives. The next card has the full treatment.
Tested empirically via random-seed ablation. If reseeds produce the same Y for the same M, the assumption is consistent with the data.
Because the CoT and the answer read from the same hidden state, the errors $U_M$ and $U_Y$ are correlated. We make that correlation a single parameter and use a probit (latent-Gaussian) outcome so it enters cleanly, the standard move in the Imai, Keele, and Tingley (2010) framework:
Conditioning on the observed $M$ makes $U_M = M - \gamma X$ known, which gives an exact observed-data likelihood at any fixed $\rho$. The sweep fits the de-confounded coefficients on a grid of assumed $\rho$ and reads off the natural effects. On synthetic data with true $\rho = 0.5$, assuming ignorability reports a faithful path of 0.43 against a truth of 0.21 (an overstatement of 0.22 that does not shrink with $n$), the sweep recovers the truth at the true $\rho$, and the faithful path stays positive for every $\rho$ in $[-0.6, +0.7]$.
One faithfulness number per model is the wrong unit. Faithfulness varies by prompt ($j$), and some prompts have far fewer usable traces than others. Each prompt gets its own coefficients drawn from a population, with a non-centred parameterisation for friendly NUTS geometry:
The population mean $\mu_\beta$ is the model-level faithfulness slope with the right
uncertainty; $\tau_\beta$ says how much prompts disagree. A prompt with five traces
is pulled toward the global mean instead of swinging wildly on its own. On synthetic
data the model recovers $\mu_\beta$ inside its 95% credible interval and beats
no-pooling on per-prompt error. A full pm.LKJCholeskyCov covariance over
$(\alpha_j, \beta_j, \gamma_j)$ is the natural extension; the independent-scale
version here is the validated v1.
Everything in this repo runs on a laptop CPU in under a minute. Scaling to thousands of intervention samples across hundreds of prompts is the planned next step, with a decoupled architecture:
Naive text edits on black-box APIs cause distribution shifts that destroy the causal estimate. The plan uses two intervention regimes matched to model access: