How sure can we be that a number means what it says?

Projects

• live

  Agent Reliability Calculator

  Shows why adding more verifiers stops helping after a point and why retrying an agent buys less than the usual formula promises. The defaults come from 131,332 real agent runs on four public benchmarks.

  browser tool4 public benchmarks
  Live Repo
  Details

  What it does

  It has four blocks. The first shows the reliability ceiling of a chain of steps with verifiers. The second compares naive retries with real ones. The third compares strategies by total cost. The fourth measures the repeat correlation (ICC) from your own CSV in the browser, and the file is never uploaded.

  What the data says

  Retries of the same agent on the same task are strongly correlated, with a median ICC of 0.49 to 0.67 across the corpora. A model fit on part of the data predicts the rest within 0.3 to 1.5 points, while the independence formula misses by 14 to 58.

  Honest limits

  ICC is not a universal constant. Across domains and models it ranges from 0.03 to 0.83, which is why you can measure your own.

• study

  Honest agent scores

  Asks whether broken benchmark tasks or repeated runs change which AI agent comes out on top. On 3,300 Terminal-Bench 4.0 runs the answer was mostly no. Broken tasks can still make agents look different in reliability when they are not.

  reproducibleTerminal-Bench 4.0
  Repo
  Details

  What it checks

  Two common claims in AI agent evaluation. One is that benchmark defects hide the true leader, the other is that ranking by reliability on repeats (pass^5) reorders agents. The data are public per-trial results for 10 configurations on 66 tasks with 5 trials each.

  What the data says

  Neither change moves the leader beyond noise. Broken tasks can create false differences in reliability between agents. With 66 tasks, two agents within about 10 points of each other are not reliably ordered, and about 500 tasks are needed to resolve 5 points.

  Honest limits

  The test has low power against a shift of one or two places. The analysis plan was written before each analysis, and exploratory results are reported separately. A small tool runs the same checks on your own leaderboard.

• live

  The Signature Explorer

  Explains digital signatures from the one in your crypto wallet to the post-quantum ones. You can sign a message, break it and even forge a signature yourself. Every fact on the page links to its original source.

  interactive explainerpost-quantum
  Live Repo
  Details

  What you can do

  Sign a message, tamper with it and watch verification fail. Reuse an ECDSA nonce on purpose and recover the private key from two signatures. Sign twice with one Lamport key and forge a message nobody signed, checked by the real verifier.

  Where it goes

  From there it follows the hash-based path through XMSS and FORS to SLH-DSA, standardised as FIPS 205. It runs entirely in the browser with no backend, and there are exercises for teachers.

  Honest limits

  An exercise completion code proves that a valid solution exists for that seed, not who produced it. The exercises are meant for practice, not proctored assessment.

How I work

I write down the analysis plan before I run the numbers. If the result is negative, I publish it anyway. Every figure in my projects can be traced back to where it came from.

1. 01
   Ask the question
2. 02
   Write the plan first
3. 03
   Measure on public data
4. 04
   Try to break it
5. 05
   Publish, even if negative