One in twenty

A gift, then quietly taken back

I have spent a lot of my time learning statistics on my own, and of everything I have picked up that way, the one thing I keep circling back to is not the p-value itself but the little line we draw across it at 0.05.

Almost nobody who uses a p-value is confused about what it is; they can give you the definition. The thing that confused me for the longest time is different. The introductory books teach that 0.05 is the significance level, you compute a p-value, check whether it sits below 0.05, and call the result significant or not, and I do understand why they teach it that way, because one concrete number makes the idea land the way a worked example always does. But it sets a quiet trap. Sooner or later that student runs into the fact that 0.05 is not the only cutoff you could use, and more often than not it is the wrong one for the question actually in front of them, and nobody ever told them the number was up for debate in the first place.

Here is the pattern I keep running into. At first that 0.05 line is handed to you as a gift, a clean cutoff, a number that makes learning easier, and if you cross it your result is “real.” Then you go one level deeper and the same 0.05 gets quietly taken away. In Statistical Rethinking, in my internships, in the papers I read, the line keeps moving, it is field-dependent and context-dependent and half the time nobody explains why, and yet a lot of the material aimed at students still treats it as a law of nature.

None of this is new, and I should say that plainly. People have been writing about that little line in books, posts, and papers for years, and even Fisher, who did more than anyone to spread the 0.05 convention, said later on that treating it as a fixed level was a mistake, and that the right cutoff should really depend on the task. So this is not me making a new case, it is more of a collection, the observations I have gathered from all of those places and a few of my own, pulled into one spot. My claim, which I will try to earn over the rest of it rather than just assert, is that the 0.05 line, the hard border between “significant” and “not significant,” is the wrong idea. Not because the p-value is broken, the number underneath is fine, but because taking a continuous measure of evidence, chopping it at one arbitrary point, and then treating which side you landed on as the finding throws away almost everything that actually matters. A lone p-value is uniquely useless, and yet the very same number turns genuinely interesting the moment you set it back next to the effect size and the context.

The clearest way I have seen to show why one fixed 0.05 is wrong is a courtroom. A court does not use a single standard of proof for every case: a criminal trial asks for proof beyond reasonable doubt, while a civil suit only needs the balance of probabilities, because the cost of being wrong is so different in the two. Hold that picture and a p-value becomes a piece of evidence, the word “significant” becomes a verdict, and 0.05 becomes a single fixed sentence a judge would read out no matter what the case in front of them was. The evidence is worth keeping. It is that one-size-fits-all sentence, the same bar for a parking ticket and for a murder, that I think we should stop trusting.

A dial someone turned into a switch

Start with the null hypothesis, H₀. In court it is the presumption of innocence, so you assume nothing is going on: no effect, no difference, the coin is fair, the drug does nothing. You do not have to believe it, you just adopt it as the baseline story, and then you ask how strange your data would look if that story were true.

That question has a precise answer and it is the p-value, and the cleanest way to hold it is as a count, not as some abstract measure of how likely the truth is. Imagine the null is true and you run the same study a thousand times over, drawing fresh data each time. The p-value is the share of those runs that come back with data at least as extreme as the run you actually got. A p of 0.05 just means about fifty of those thousand runs, one in twenty, would look at least this surprising, which is where the title of this piece comes from. People write it in symbols as P(data this extreme | H₀ true), but that count is the thing to keep in your head. Two more things are worth holding onto: it is a continuous number between 0 and 1, a dial and not a switch, and a small p only means that in a world where nothing is going on, data like mine would be an uncommon accident, which is real evidence against the null but not proof of anything on its own.

Drag the marker below. It is the test statistic, how far your result sits from what the null expects, measured in standard deviations. The curve is the world the null predicts, where results near zero are common and results far out are rare. The shaded tail is everything “at least this extreme,” and its area is the p-value. Watch how smoothly the number slides as you drag, because that is really the whole problem with 0.05: it is one arbitrary spot on a continuous slide, and the data just to either side of it is barely different at all.

The one mistake everything else descends from

There is one more slip worth slowing down for, and it catches people who know the definition perfectly well. The p-value is P(evidence | innocent). What we actually want to know is the other way round, P(innocent | evidence), meaning given this result how likely is it that nothing is really going on. Those two are not the same number, and swapping one for the other even has a name. It is the prosecutor’s fallacy, and it is called that because it is a courtroom error in the first place.

A prosecutor tells the jury that the chance an innocent person’s DNA would match is one in a million, so there is only a one-in-a-million chance the defendant is innocent. That is flatly wrong. The first number is the chance of the evidence assuming innocence, the second is the chance of innocence given the evidence, and turning one into the other quietly ignores how many people were in the pool to begin with. A p-value of 0.03 does not mean there is a 3% chance the null is true, or a 97% chance your effect is real. It never said that, we just keep hearing it say it.

So a bare p-value answers exactly one question, how surprising my data is if nothing is going on, and it stays quiet on every question we actually care about, like whether the effect is real, whether it is large, whether it matters, whether it would happen again. Those silences are not a flaw in the number, they are the number being honest about its job. The trouble only starts when we mistake the silence for an answer.

Where 0.05 actually came from

If 0.05 were a law of nature you would expect a derivation somewhere, a cost-benefit calculation, a proof, something. There isn’t one. The line is a convention that slowly hardened into a commandment, and tracing how that happened is the fastest cure I know for treating it as sacred.

Ronald Fisher usually gets the credit, but he inherited the number more than he invented it. The historians Michael Cowles and Caroline Davis traced a “three probable errors” rule of thumb, which is numerically close to the two-standard-deviation line we still use, circulating among statisticians well before Fisher published anything, back around 1908 and building on Edgeworth in the 1880s. Something like a 5% habit was already in the water.

In his 1925 handbook Fisher framed 5% as exactly that, a convenience. It was, in his own phrasing, simply convenient to draw the line at one-in-twenty for a researcher doing arithmetic by hand, a practical cutoff rather than a claim about the fabric of the world. A year later, in a 1926 paper on field experiments, he made the point that it was one option among several, so if one-in-twenty did not feel strict enough for a given case you could just as easily use one-in-fifty (2%) or one-in-a-hundred (1%). The number was always meant to flex with the stakes.

In his 1956 book he wrote, in plain terms, that no working scientist has a single fixed level of significance that they apply to every experiment year after year, and that you weigh each case on its own evidence. That is about as direct a repudiation of a permanent 0.05 rule as he ever put in print. Richard McElreath reads that same 1925 passage the same way in Statistical Rethinking, as codifying a convention that was already in wide use rather than justifying why 5% in particular is right, and his verdict on the threshold is blunt, that it is purely conventional and there is no reason to expect it to be optimizing anything at all.

The rigid version won out anyway, and the reason is that a second framework got welded onto Fisher’s. Jerzy Neyman and Egon Pearson recast testing as a decision procedure, where you fix an error rate, call it α, before you see the data, then mechanically accept or reject, and you judge the whole apparatus by its long-run error rate rather than the evidence in any one study. They also added the idea of the missed effect, the false negative, which Fisher’s evidence-only view did not really have. Fisher thought the whole thing was industrial quality-control thinking smuggled into science, and he said so, bitterly, for the rest of his life.

The number we all use is the wreckage of that fight. The everyday ritual, where you compute one p-value, compare it to a fixed 0.05, and declare “significant,” treats a single result as if it were both Fisher’s continuous measure of evidence and Neyman and Pearson’s pre-committed accept/reject decision at the same time, and it is cleanly neither. Gerd Gigerenzer calls this hybrid the “null ritual,” and its authority is really borrowed from two philosophies that never actually agreed with each other. The verdict sounds firm because we are quietly running two incompatible courts through one gavel.

Why a lone p-value misleads

Suppose we do everything by the book, one honest study, a clean p < 0.05, a “significant” result. How often is that verdict actually right. The uncomfortable answer is that it depends entirely on things the p-value never shows you, and it can be wrong far more than 5% of the time.

The reason is base rates, the same thing the prosecutor ignored. A verdict of “significant” sweeps up two very different kinds of study, real effects that got correctly caught and null effects convicted by bad luck, and the share of your convictions that are false depends on how many true effects were in the pool to begin with. Test a lot of long-shot hypotheses and even a well-run filter hands you a courtroom full of wrongful convictions.

Play with it below. You set how often a tested hypothesis is really true (the base rate), how good your study is at catching real effects (its power), and your false-alarm rate α. The array shows 1,000 hypotheses sorted into the four outcomes, and the number that should unsettle you is the false-discovery rate, meaning among everything you called “significant,” what fraction is actually a null effect wearing a guilty verdict.

McElreath runs essentially this calculation with a stricter setup than the figure’s default, a hypothesis true only 1 in 100 times, tested at 95% power. There a “significant” result is correct only around 16% of the time, and you can land there yourself by dragging the base rate down to 1% and power up near 95% above. Tightening the threshold to 1% barely rescues it, you get to roughly a coin flip, and the biggest lever by far is not a stricter line, it is testing hypotheses that stand a decent chance of being true in the first place. David Colquhoun ran a similar calculation and got a false-discovery rate of at least 26% for a result sitting right at p = 0.05, though, exactly as the figure warns, that number rides on his assumptions and other reasonable assumptions move it.

“Not significant” is not “nothing there”

The mirror-image error is just as common and a lot quieter. A study comes back p = 0.12 and gets written up as “no effect,” but failing to convict is not the same as proving innocence. Absence of evidence is not evidence of absence, and a small, underpowered study can miss a real and important effect simply because it never gathered enough data to see it. The honest reading of a non-significant result is usually “we can’t tell yet,” not “there is nothing here,” and a whole section below is about people who paid for that confusion.

This, by the way, is exactly why the careful phrasing is always “reject the null” or “fail to reject the null,” and never “accept the null” or “accept the alternative.” That wording is doing real work, because failing to convict is not the same as declaring innocence, and it is honestly one of the few corners of this whole subject where the standard teaching gets the nuance exactly right. The irony is not lost on me: that careful reject-or-fail-to-reject language probably trips up a first-time learner more than anything else, and yet it is taught precisely as it should be, while the one thing that really is worth flagging, that the 0.05 does not have to be 0.05, mostly gets left out.

There is an even subtler trap here. If one study lands significant and another does not, it is tempting to say the two disagree, but the difference between “significant” and “not significant” is not itself statistically significant, and two studies can be perfectly compatible and still fall on opposite sides of the line. The line invites us to see a conflict that the evidence does not actually support.

A garden of forking paths

Here is the one that surprised me most, because it needs no villain at all. Hand an honest researcher a single dataset that has nothing in it, pure noise, no real effect. To analyse it they have to make a string of small, reasonable decisions along the way: which outcome to treat as the main one, whether to adjust for age or not, whether to drop a couple of odd-looking data points, whether to look at everyone or just one subgroup. Each decision on its own is completely defensible, the kind of thing a reviewer would nod at. The catch is that all of those decisions multiply into a big tree of possible analyses, and even on pure noise, if you laid the whole tree out, some of its branches would land under 0.05 just by luck.

You might think that is fine as long as the researcher walks one branch and reports only that one, and this is the subtle bit Andrew Gelman and Eric Loken pointed out. The researcher really does pick a single path, honestly, and never runs a hundred analyses. But which path they pick depends on how the data happened to come out, so if the noise had landed a little differently they would have made different calls and walked a different branch. Across all the branches they could have ended up on, the chance of stumbling into a “significant” result is far higher than the 5% the final p-value quietly claims. It is multiple testing without anyone ever running multiple tests, which is also why you cannot simply correct for it: nobody, not even the researcher, knows how many paths were really in play. Click through the tree below. Nothing real is hiding in the data, and you can still watch how many perfectly defensible routes cross the line.

The dance of the p-values

Last, the thing that finally broke my trust in reading a single p-value too closely. This time there really is a genuine effect, one that exists in the world, and the sample size is fixed too. The only thing that changes from run to run is the random sample you happen to draw. Run the study once and you get a p-value. Run the exact same study again on fresh data and you get a different p-value, and not a little different, it jumps around across orders of magnitude. Geoff Cumming put a number on how wild this is: a study that lands at exactly p = 0.05 could, if you simply repeated it, come back anywhere from about 0.00008, which is wildly significant, to about 0.44, which is nowhere close, all from the very same true effect. So agonising over whether your p came out 0.049 or 0.051 is a little absurd, because a rerun could have handed you almost anything. The figure below draws that scatter, one dot per repeated study, so you can watch the p-value refuse to sit still.

Put those four together, the base-rate trap, the “not significant” trap, the forking paths, and the dance, and a single p-value stripped of its context starts to look less like evidence and more like a coin that only remembers how it landed. It is not meaningless, it is just nowhere near enough on its own to convict anyone.

This is not just a methods quibble

It would be easy to file all of this under academic hygiene, but it is not that. The specific habit, where you run a test, check whether p clears 0.05, then declare an effect either “real” or “not there,” has a documented history of getting the wrong answer at the moments it mattered most. The recurring mistake is always the same one from section 02, reading “not significant” as “no effect,” when the honest reading was “we do not have enough data yet.”

The fix these cases point to is not a stricter threshold. It is dropping the reflex to collapse a result into “real” or “not there” at all, and reporting the effect size and its uncertainty instead. That is harder work than checking whether a number cleared 0.05, and it is also what would have caught the shoulder-paving effect and the CAST mortality signal a good deal earlier.

The threshold moves, on purpose

If 0.05 were a natural constant, every field would use it. They don’t, and the differences are the clearest proof that the line is a choice about stakes, not a fact about truth. Think of each field as a court with its own burden of proof, set by what it costs to be wrong.

Particle physics wants near-certainty before it will say the word “discovery,” because a false discovery there is a public humiliation. Its bar is five sigma, a one-sided p-value of about 2.9×10⁻⁷, roughly one in 3.5 million. That is more than five orders of magnitude stricter than medicine’s everyday 0.05, since 0.05 over 2.9×10⁻⁷ is a factor of about 170,000. There is a myth worth killing while we are here, which is that physics also has a looser “three sigma” level, around p ≈ 0.0013, but that one is explicitly “evidence worth a closer look,” never a discovery. The half-remembered idea that physicists discover things at p around 0.003 confuses the “keep watching” bar with the “announce it” bar.

Genomics goes stricter still, and for a very concrete reason. Run one test at 0.05 and you accept a 1-in-20 chance of being fooled by noise. Run twenty independent tests at 0.05 and you should now expect about one of them to come back “significant” even if nothing real is going on anywhere, and the chance that at least one does is roughly 64%. The oldest fix for this is the Bonferroni correction, and it is as blunt as it sounds: if you are running twenty tests, demand 0.05 divided by twenty, so 0.0025, from each one, and the whole batch keeps its 5% false-positive rate. A genome-wide study is that taken to an extreme, testing something like a million variants at once, which is exactly why it draws the line at 5×10⁻⁸ rather than 0.05. And notice the link back to the forking paths: Bonferroni only works when you can count the tests. The garden of forking paths is the version where the tests are invisible, so there is nothing to divide by.

Meanwhile methodologists argue about the default itself. One camp (Benjamin and dozens of co-authors, 2018) proposed dropping the line for new discoveries from 0.05 to 0.005. Another, led by Daniel Lakens, replied “justify your alpha,” where the fix is not another universal number but asking researchers to defend whatever threshold they use for the specific study, which is, tellingly, Fisher’s late position all over again.

Underneath all the arguing is one idea worth stating flatly: whatever else a significance threshold is, it is a control on your false positive rate. When you fix α at 0.05 you are making a promise, that if the null is true you will let noise fool you into calling it significant no more than one time in twenty. That is the whole job of the number. And it is a trade-off, because every test balances that false alarm (a Type I error, convicting the innocent) against a miss (a Type II error, letting the guilty walk). Push α down and you convict fewer innocents but miss more real effects, unless you also gather more data. The figure below is that trade-off, and you can grab it. Drag the cutoff and watch the false-alarm and miss regions trade against each other.

And there is a threshold beyond the threshold that the ritual skips entirely. A result can be statistically significant and still be far too small to care about, because with a big enough sample a trivial difference will eventually cross any line. So careful fields pair the significance test with a pre-declared effect size that would actually matter, the minimal clinically important difference (MCID) in medicine, or the smallest effect size of interest (SESOI) in psychology. A drug that lowers blood pressure by a statistically airtight 0.4 mmHg has cleared the first bar and failed the one that counts. Significance is necessary, but it is never sufficient.

Better tools, and a better mindset

If the lone p-value is a weak witness, the obvious next question is what to call instead. None of the alternatives are magic, and I want to be honest about that, but each one fixes a specific blind spot, and together they add up to a saner way to read evidence.

Actually proving “no meaningful effect”

Ordinary testing can never support “there is no effect,” because failing to reject the null just means you did not gather enough to rule zero out. But sometimes proving innocence is exactly the question you have, like whether this generic drug is equivalent to the brand, or whether this change is harmless. The tool for that is equivalence testing, usually done as TOST, two one-sided tests. You first set two bounds around zero, defining the band of effects small enough to count as “no meaningful difference,” and those bounds come from your SESOI, a judgment about what matters, not from the data. Then you run two tests, one asking whether the effect is reliably above the lower bound and one whether it is reliably below the upper one, and if both pass you have earned the right to say the effect is negligibly small, which is a claim plain significance testing can never make.

There is a clean shortcut hiding here, that running TOST at α = 0.05 gives the same verdict as checking whether the 90% interval (note: 90%, not 95%) sits entirely inside your bounds. Which points at the deeper move, just report the interval.

Compatibility, not confidence

A confidence interval already carries both of the things a lone p-value cannot, whether zero is plausible and whether an effect big enough to matter is plausible. The trouble is the word “confidence,” which invites two misreadings, treating the interval as pass/fail (does it touch zero?), and saying “there is a 95% probability the true value is in here,” which is not what it means. Sander Greenland and colleagues suggest renaming it a compatibility interval, the range of effect sizes reasonably compatible with your data. Read that way it is a list of the stories the evidence tolerates, a spectrum to read rather than a verdict.

Surprise, measured in coin flips

One reason p-values feel slippery is that the scale is deceptive, so p = 0.001 feels ten times as surprising as 0.01, but it is not. A cleaner scale is the surprisal, or S-value, S = −log₂(p) bits, which you can read as how many heads in a row your data is as surprising as, if the null were true. A p of 0.05 is about 4.3 bits, so roughly four or five heads in a row, which is mildly surprising and not proof of anything. Drag the slider and feel how slowly surprise actually accumulates.

And a fair word on going Bayesian

The Bayesian alternatives, Bayes factors and posterior probabilities, answer the question we actually wanted all along, which is, given the data, how the hypotheses compare. That is a real improvement over the transposed conditional. But I want to be careful not to oversell it, because the pitch is often a bit dishonest. A Bayes factor requires a prior, and the prior is a judgment call. Bayesian methods do not remove the subjective choice buried in the analysis, they make it explicit and put it in the open, which is better, but it is a trade of one judgment for another rather than a subjectivity-free machine that hands you the truth. Anyone selling it as that machine is doing the same overselling that got p-values into trouble in the first place.

Earn the thesis, carefully

I opened by calling the 0.05 line the wrong idea, and I owe you a more precise version of that now, because the sloppy version is wrong and I do not want to commit the very sin this piece is about.

So, to be exact about it, the thing I think is wrong is not the p-value but the hard 0.05 border, the reflex to take a continuous number, ask only whether it landed above or below one fixed point, and report which side as the finding. That is my opinion, and I have tried to earn it over the base-rate trap, the dance, the forking paths, and the fields that all pick different lines. The scope matters though, because a p-value that is reported as the continuous thing it is, next to an effect size and a compatibility interval, is genuine and useful information. The problem was never really the number, it is the line we nailed across it.

That is also, roughly, where the profession has landed. The American Statistical Association’s 2016 statement did not ban p-values, it pushed back on the misuses, that p is not the probability the hypothesis is true, that thresholds should not drive conclusions on their own, and that p says nothing about effect size. The 2019 Nature comment signed by hundreds of researchers went further and argued for retiring the “significant / not significant” dichotomy altogether, and note that this is not the same as abandoning p-values, it is about dropping the bright line that invites all this abuse wherever it happens to be drawn.

That is the part I feel most, as someone still fairly early in learning this. The 0.05 line gets handed to students as a kindness, a way to make an overwhelming subject tractable, but the kindness is a lie of omission, and the correction, that the line is conventional and contextual and half the story at best, usually arrives years later if it arrives at all. I would rather it were taught with its context intact from the beginning, as evidence rather than a verdict, as one witness rather than the whole jury. The number deserves that, and so do the people who will spend their careers trusting it.

So keep the three apart, the evidence, the verdict, and the fixed sentence nobody really justified. The evidence is worth reporting, carefully, with its effect size and its interval next to it. It is the sentence, one number read out over every case regardless of the stakes, that we should stop treating as the answer.

Every metaphor, its textbook name

A cheat sheet for when you take this to a paper or a stats course. The left column is what I have been calling things, the middle is what you will see them called out there, and the right is a one-line reminder of the job each one does.