System One · benchmark v1 reference
System One.
Benchmark v1 reference.
Ranking coding agents instead? LLM setups — harness, model, thinking →
Typed decisions
The original 49-question benchmark.
The original dataset contained 23 scenarios and 49 questions. Several models reached 98%, so it stopped separating the strongest endpoints. These historical rankings and the phishing workload below remain available as a reference. Benchmark v3 (and v2) use new datasets; their scores cannot be compared directly with v1.
Real workload
The anti-phishing classifier, unchanged.
Sixteen adjudicated emails through the service's own eval harness and its typesafe_only arm — zero code changes, only the backend swapped by environment. The pipeline, not the model, makes the final risk decision.
Ledger
Every System One v1 run, filterable.
The system1-legacy and historical phishing-eval rows. Click a column to sort, click a model to open its report. Green marks the best score per suite in view. Coding-agent runs live on the LLM ranking page.
Reproduce it
Reproduce benchmark v1.
One endpoint, any backend: configs/s1_gateway.py on :8123. Swap the backend by environment variable; the application never sees the change.
./bench validate --suite system1-legacy