Loading…

Small sample (30 outcome cases, 170 record-validation cases), one run. Treat as a screening test, not a benchmark.

Dot = accuracy on the base pass; whisker = 95% bootstrap CI (the same method as report.md). Dashed line = the 90% guard bar. n per family shown beside each name.

Each line joins the same underlying records shown two ways. p = exact McNemar test on the pairs where only one version was right.

Records Jev had already flagged as invalid, before and after the manipulation. A flip means Jev changed its answer to “valid”.

Accuracy on the 30 outcome cases under three option-naming conditions. “Misleading” permutes names against their rule descriptions.

Reliability: Jev’s stated P(valid) against how often records were actually valid, in 10 bins (record validation, n=170). Dot size shows bin count; the diagonal is perfect calibration.

Selective automation: answer only the most confident cases first. x = share of cases auto-decided, y = accuracy among them.

Every one of the 735 calls is here to inspect

Filter by family, pass and right/wrong. The state shown is the base case; the opinion, key-shuffle and naming passes alter how it was presented (see Method).

Case	Family	Pass	Jev	Expected

What this test can and cannot tell you

Small samples. 10–30 cases per family; the confidence intervals are wide, and a family at 100% could still be imperfect.
One run. A single pass of each condition, so the intervals describe the choice of cases, not run-to-run variation (the re-run pass is the only repeat).
Alpha API. OpenRouter’s Decisions API is alpha; behaviour may change. The exact model version is stated at the top.
Labels computed by rules. Every correct answer comes from code in cases.py, not from people. A rule that is wrong would be wrong here too.
Synthetic cases. They test the rules we wrote down, not every real failure a pipeline will meet.

Passes: base asks each case once; rerun repeats it; key_shuffle reorders the state’s keys; opinion appends “I double-checked… looks correct to me” to invalid records; neutral_names and misleading_names rename or permute the outcome options.

SUMMARY.md · report.md · Data: summary.json, results.json, cases.json, run_meta.json, stats.json, cost.json · Code: cases.py, run.py, report.py, make_site.py

Regenerate with python make_site.py results-live. Prose on this page was written for the run shown above; re-read it after a new run.