System One · benchmark v3
System One.
Benchmark v3.
Ranking coding agents instead? LLM setups — harness, model, thinking →
Typed decisions
Decisions, not arithmetic.
On benchmark v2, about three quarters of the gap between Jev and reasoning models came from multi-step calculation: time-zone conversion, cent rounding, stock sums. Benchmark v3 hands over those quantities already computed and tests the decision on top: 200 scenarios, one combined decision-and-reason question each, 200 calls per model. Typed decision services, local models and thinking models answer the same prompts, at one sample and concurrency 1. v3 is the default system1 suite; v2 remains as a reference.
Where models differ
Accuracy by decision family.
Twenty scenarios per family, so one miss moves a cell by 5 points. Read gaps under about 15 points as indicative only. Green marks the best score in each row. Families where every model scores near 100% no longer separate them.
Dataset v3
200 scenarios. 200 calls.
Ten families of 20 scenarios. Facts arrive in prose: customer emails, chat transcripts, staff notes. Models must tell claims from confirmations, notice when a case cannot be decided yet, and ignore instructions planted in untrusted text. Every question has listed options, and each answer names the decision and its reason together. Six families state each case twice in different wording; a model must answer both versions correctly.
One sample per question gives 200 responses per model. With 200 items a 95% interval is about ±5 points near 85%, so smaller margins are noise. Paraphrase pairs are correlated, so intervals describe scored attempts. Read the dataset design →
Ledger
Every benchmark v3 run, filterable.
Runs on the 200-question system1 v3 dataset. v2 results and v1 and phishing results have their own pages. Click a column to sort, click a model to open its report. Green marks the best score per suite in view. Coding-agent runs live on the LLM ranking page.
Reproduce it
Run benchmark v3 on your endpoint.
One endpoint, any backend: configs/s1_gateway.py on :8123. Swap the backend by environment variable; the application never sees the change.
./bench run --suite system1 --samples 1 --concurrency 1