System One · benchmark v2 · reference
System One.
Benchmark v2.
Ranking coding agents instead? LLM setups — harness, model, thinking →
Typed decisions
Fresh measurements on a harder dataset.
Benchmark v2 contains 200 distinct scenarios and 400 scored questions. Each question supplies the facts and rules needed for a short decision. Jev, local Nimble-9B and local Kev-4B were measured through the same gateway, with two samples per question and concurrency 1. v2 has been superseded by benchmark v3: about three quarters of the gap between decision models and thinking models here came from multi-step arithmetic, which v3 precomputes. The v1 rankings remain a separate historical reference.
Dataset v2
200 scenarios. 400 decisions.
Ten equally sized families test exceptions, close boundaries and conflicting evidence. The dataset has 380 multiple-choice questions and 20 open code-extraction questions, with two scored questions per scenario. All necessary rules and facts are supplied.
Two samples per question produce 800 responses per model; repeated samples do not add new scenarios. Questions sharing a scenario are correlated, so confidence intervals describe scored attempts. Read the dataset design →
Ledger
Every benchmark v2 run, filterable.
Runs on the 400-question v2 dataset, now system1-v2. Historical v1 and phishing results have their own page. Click a column to sort, click a model to open its report. Green marks the best score per suite in view. Coding-agent runs live on the LLM ranking page.
Reproduce it
Reproduce benchmark v2.
One endpoint, any backend: configs/s1_gateway.py on :8123. Swap the backend by environment variable; the application never sees the change.
./bench run --suite system1-v2 --samples 2