The cheapest decisive experiment: how we pick the next test
Published September 4, 2026 · SynthoCore Automation Inc.
A defect lands and two explanations arrive with it, each perfectly consistent with what you have seen. The request fails; one engineer is sure it is the cache, another is sure it is the clock. Both stories fit every symptom on the table, which is exactly why neither can be waved away. The question that decides how long the bug lives is not “which explanation is right?” — you cannot yet know — but “what is the next thing I should do?” And the instinct almost everyone brings to that question is the wrong one: reach for the experiment that would confirm the hypothesis you already favour. The better discipline is to reach for the experiment that would settle the disagreement whichever way it turns out, and among those, the one that costs the least to run.
We call that the cheapest decisive experiment, and choosing it is a ranking problem, not a guessing one. Rank candidate experiments by cost-to-decisiveness — how much of the disagreement a result eliminates, divided by what it costs to get — and run the top of that list. How confident you feel in a hypothesis does not enter the ranking at all, because confidence is the thing under test.
Decisiveness is a property of the experiment, not the outcome
An experiment is decisive when every outcome it can produce points somewhere different. Line the live hypotheses up in a row; a decisive test draws a line through that row so that each side falls into a distinct result. Whatever happens when you run it, some hypotheses are ruled out and others survive — the space is smaller than it was, no matter which way the result lands. That last clause is the whole test. Before you run anything, ask what you would conclude from each possible result. If a “pass” would eliminate one story and a “fail” would eliminate the other, the experiment is decisive and worth its price. If both a pass and a fail leave you holding the same set of explanations, the experiment tells you nothing, however satisfying it is to watch.
Notice what this buys you. A decisive experiment cannot come back “inconclusive” in the way that matters, because you agreed in advance what each result would mean. You are not hoping for a particular answer; you are arranging for any answer to move you forward. That is why the feeling of confidence is beside the point. The value of the test is fixed before it runs, by its structure, and does not depend on which branch you were rooting for.
Cost is the tiebreaker, and it is usually large
Decisiveness gets an experiment onto the shortlist; cost decides which one you run first. Among tests that would each split the hypothesis space, prefer the one that is cheapest to set up, fastest to return, and least likely to disturb the thing you are studying. A grep through existing logs beats adding a metric; adding one log line and replaying a saved request beats standing up a reproduction; a reproduction you already have beats one you must build. The ordering is by information per unit of effort, and the effort term swings the ranking more often than newcomers expect, because two experiments that are equally decisive can differ by an order of magnitude in what they cost to run.
This is where the ranking earns its keep. The hypothesis you privately favour often suggests an expensive, elaborate test — instrument the whole path, reproduce the exact conditions, watch it happen. A rival hypothesis you find less likely might be refutable in thirty seconds with a query you can already write. Run the cheap one first. If it splits the space, you have spent thirty seconds; if it doesn’t, you have lost almost nothing and you reach for the next item on the list. Feelings would have sent you to the costly test; the ranking sends you to the decisive-and-cheap one.
The anti-pattern: the confirmation run
The failure mode to name and avoid is the confirmation run — an experiment that can only ever agree with you. You believe the cache is stale, so you clear the cache and watch the request succeed, and you close the ticket satisfied. But ask the decisive question: what would you have concluded if it had failed? Nothing you would have accepted — you would have said the cache had already repopulated, or the wrong layer was cleared, or the timing was off, and reached to clear it again. A test whose failure you would explain away is a test you were never really willing to lose. It has one reachable outcome, and an experiment with one outcome carries no information; it launders a prior belief through the ceremony of running something.
Confirmation runs are seductive precisely because they usually “work.” The system is complicated enough that nudging the thing you suspect often changes the symptom, and the change reads as vindication. The guard against it is a single question asked before you run anything: which result would change my mind? If you cannot name a concrete outcome that would send you to the other hypothesis, you are not about to run an experiment. You are about to perform one.
A worked shape: one request, two reasons, one line that splits them
Here is the generic situation, stripped of any particular system. A request fails, and it fails identically under two different causes — the caller receives the same error, the same status, the same unhelpful message, whether the fault is a permission the caller lacks or a dependency that was unreachable when the request ran. Same surface, two roots. Every symptom you can see from the outside is consistent with both, so no amount of staring at the response will separate them, and the two fixes could not be more different: one is a policy change, the other is a retry-and-resilience change. Guess wrong and you ship a fix for a bug you do not have.
The confirmation run here is to assume it is permissions, widen the caller’s access, replay the request, and watch it pass — which it may, for reasons having nothing to do with permissions, because the flaky dependency happened to answer that time. The decisive experiment is to find or add the one observation that the two causes disagree on. The outward failure is identical; the internal reason is not, and it is written down at the moment the decision is made.
The point is not the log statement itself; it is that a single, cheap observation was chosen because its two possible values fall on opposite sides of the disagreement. One replay produces exactly one of the two lines, and whichever appears, half the hypothesis space is gone. Contrast the confirmation run, which could only ever have produced “it passed” and would have taught you nothing about why it failed. The decisive line costs about the same to add and cannot come back empty. That is the trade you are always looking for: the smallest instrument whose readings the competing explanations do not agree on.
The through-line
When explanations compete and the evidence so far fits all of them, the next move is not to gather support for your favourite — it is to run the cheapest experiment that would settle the matter whichever way it lands. Rank candidates by how much of the disagreement each result eliminates against what it costs to get the result, and start at the top. Refuse the confirmation run by insisting, before you touch anything, on naming the outcome that would change your mind; if there isn’t one, you have not designed an experiment. Two identical failures with two different causes are separated not by conviction but by the one observation their causes disagree on. Find that observation, make it cheap, and let the result — not your confidence — decide what you believe next.
If this is the kind of rigour you want in how your systems are built and reviewed, get in touch.