Why adding verifiers hits a ceiling, and why retries deliver less than 1−(1−p)k promises.
Every curve here is calibrated on 131,332 real agent runs from four corpora —
report and methodology · where each number comes from.
Runs entirely in your browser; nothing is uploaded.
Drag verifier strength all the way right — the curve hits a ceiling and stops. That ceiling is set not by how good the verifier is, but by its blind spot: the share of errors it structurally cannot see. The only way to raise it is diversity — independent verifiers with different blind spots.
Calculators usually treat retries as independent: 1−(1−p)^k. But retries of the same configuration are
correlated — either this agent can solve the task or it can't. Below, both curves side by side.
The presets come from real measurements: public Terminal-Bench 2.1 and 4.0 leaderboards (2026 models, through September 2), SWE-bench Lite and τ-bench.
Presets from our own measurements (click):
Four corpora, three domains, 131,332 runs plus 74 measurements across dozens of models (the freshest from September 2, 2026: Claude Fable 5.1 — ICC 0.51).
The same models give 0.59 on Terminal-Bench 2.1 and 0.49 on the much harder 4.0, while moving from Fable 5 to Fable 5.1 on one benchmark barely changes it (0.49 → 0.51): ICC belongs to the model × task mix, not to the model.
Validated out of sample: the model is off by 0.6–1.5 pp — 2.1× closer than the nearest reasonable alternative, and 10× closer than the naive independence formula.
⚠️ There is no universal constant. The typical range is 0.4–0.7, but across domains and models it spans 0.03 to 0.83 — wider than the spread between the corpora themselves. Measure your own (block below).
Uses the parameters from the blocks above. Compares strategies on total cost: losses from errors plus what you pay for calls. Sometimes brute force (many retries) loses to an architectural fix (a shorter chain) — here you can see it in dollars.
Cost of an error = manual triage + compensation + customer churn. Cost of a call = model and infrastructure bill for one attempt.
Don't take our number — measure yours. Load a CSV of your runs and the correlation is computed right in the browser: the file is never uploaded. Three columns are needed: task, configuration (model/prompt/agent) and outcome (1/0, true/false, pass/fail). Column names are detected automatically.
Minimum: 30 groups with ≥3 repeats. Fewer than that and the estimate gets noisy.
The model. Step error e = 1−p; a fraction b is invisible to verification (the blind spot), the rest are caught with probability r.
With k verifiers correlated at ρ: b_eff = b^(1+(k−1)(1−ρ)), e_eff = e·[b_eff+(1−b_eff)(1−r)^k], success = (1−e_eff)^N.
Simplifications: steps are independent and verification produces no false positives. Real chains are worse — errors correlate and accumulate.
Theoretical basis: arXiv:2607.13918 (July 2026).
In reliability engineering the same phenomenon is known as common-cause failure / beta-factor (IEC 61508).