Decisio

Benchmarks

What it does, measured and dated

Ask 50 questions about one document and get every answer in about a second, on one card you own. Every number below is a measurement, with its date and its source.

Evaluation cardJevBench live boardDecision Index board

Speed

Many questions about one document

On one box (one RTX PRO 6000 at 600 W, one session on 2026-10-11), Decisio 31B answered 50 questions about a 3,000-token document in 1.08 s and about a 6,000-token document in 1.73 s; Jeff-1.0-Large took 47.1 s and 114.6 s, and Quyet-1.0-Large 48.9 s and 114.8 s; neither accepts a 32,000-token document (Decisio 31B: 10.1 s). That is about 44 times (3,000 tokens) and 66 times (6,000 tokens) as long for Jeff-1.0-Large and Quyet-1.0-Large as for Decisio 31B.

On one box, the first question about a new 3,000-token document takes a median of 112 ms on H2O-Lightning-4B, 295 ms on Decisio 12B, 490 ms on Decisio 31B, and 895 ms on Jeff-1.0-Large and 895 ms on Quyet-1.0-Large.

Bar chart of the median time to answer 50 questions about one document, five systems measured in one session on one box. For a 3,000-token document: decisio Gemma 4 31B 1.08 seconds, decisio Gemma 4 12B 1.57, H2O-Lightning-4B 2.05, Jeff-1.0-Large 47.1, Quyet-1.0-Large 48.9. For a 6,000-token document: 1.73, 1.97, 2.82, 114.6 and 114.8 seconds. For a 32,000-token document only decisio Gemma 4 12B (6.76 s), H2O-Lightning-4B (9.05 s) and decisio Gemma 4 31B (10.1 s) answer; Jeff-1.0-Large and Quyet-1.0-Large are marked not supported.Bar chart of the median time to answer 50 questions about one document, five systems measured in one session on one box. For a 3,000-token document: decisio Gemma 4 31B 1.08 seconds, decisio Gemma 4 12B 1.57, H2O-Lightning-4B 2.05, Jeff-1.0-Large 47.1, Quyet-1.0-Large 48.9. For a 6,000-token document: 1.73, 1.97, 2.82, 114.6 and 114.8 seconds. For a 32,000-token document only decisio Gemma 4 12B (6.76 s), H2O-Lightning-4B (9.05 s) and decisio Gemma 4 31B (10.1 s) answer; Jeff-1.0-Large and Quyet-1.0-Large are marked not supported.

Measured by us. One box, one session: an RTX PRO 6000 at 600 W (enforced), each system on its own published server, SM clock p50 2,220 to 2,610 MHz; median of 5 or 10 documents (the count is in every cell); synthetic support tickets by the Gemma 4 tokenizer; this compares time, not accuracy.

The server was decisio v0.11.1 on vLLM 0.30.0, on the plain layout.

Each time is one request at a time; Scale says how many a card serves at once.

Each base

Choosing a base

Three models serve the same API. Each was measured by us with the public harnesses, and none of these is a board score. Choosing a model says how to pick and start each one.

  • Gemma 4 31B, the default: at or above the other two on every accuracy measure in the table, and the slowest to read a new text. One NVIDIA card, measured on 96 GB and also run on an 80 GB H100, or a 64 GB Mac.
  • Gemma 4 12B: the smaller Gemma, for a 32 GB Mac, or a laptop with Ollama.
  • Qwen3.6-35B-A3B: the quickest to read a new text, and the only one measured with images in the text.

Accuracy and time, each base

each base with its own defaults on one RTX PRO 6000, vLLM 0.30.0, 2026-10-04; higher is better, but for the times; each column links to its record

MeasureQwen3.6-35B-A3BRecord: runs/2026-10-04_gemma-base/Gemma 4 12BRecord: runs/2026-10-04_gemma-base/Gemma 4 31B (default)Record: runs/2026-10-04_gemma-4-31b/
JevBench published items, easy / standard / hard1.000 / 0.972 / 0.7391.000 / 0.972 / 0.7391.000 / 1.000 / 0.838
BANKING77, macro-F10.7460.7290.785
CLINC150+OOS, macro-F10.8220.8710.902
GPQA Diamond, accuracy0.5100.3780.520
MMLU-Pro, accuracy0.6130.5490.694
Intent heads, BANKING77 / CLINC1500.840 / 0.9120.832 / 0.9080.844 / 0.970
Image input, ImajevBenchRecord: runs/2026-09-27_image-input/0.791n/mn/m
New 300-token state49.9 ms53.6 ms80.7 ms
1,000-token state, cached20.2 ms24.8 ms31.7 ms

n/m is not measured.

Intent heads use labelled examples, so they are not comparable with zero-shot systems.

Latency is server time on decisio 0.8.1's served defaults, at a 585 W power limit (the default is 600 W).

Gemma 4 12B and Qwen3.6-35B-A3B on the same items, with JevBench v1.5, one RTX PRO 6000, vLLM 0.30.0, 2026-10-04

What drives the choice, measured on one RTX PRO 6000 Blackwell in one session, each base with its own defaults, paired over the same items (95% bootstrap intervals; runs/2026-10-04_gemma-base/):

Gemma 4 12B against Qwen3.6-35B-A3B on the same items
MeasureGemma 4 12BQwen3.6-35B-A3BGemma minus Qwen
JevBench v1.5 open-set reading, I_open (equal types)64.449.4+15.0 [+7.0, +23.6]
JevBench v1.5, yes/no / score49.0 / 63.713.7 / 57.2
Yes/no answers between 0.20 and 0.80 (74 items)15%39%
JevBench, published items, correct0.0 points [-4.3, +4.3]
Decision Index accuracy, CLINC150+OOS+4.5 [+3.6, +5.5]
Decision Index accuracy, BANKING77-1.4 [-2.7, -0.1]
Decision Index accuracy, GPQA Diamond-13.6 [-21.7, -5.6]
Decision Index accuracy, MMLU-Pro-6.4 [-7.2, -5.5]
1,400-item suite, accuracy0.7350.770-3.5 [-5.5, -1.6]
One question on a new 3,000-token state, server time (0.8.1's served defaults, the engine in the server's process; one RTX PRO 6000 at 585 W, AMD Ryzen Threadripper 9960X; one session, runs/2026-10-06_latency-585w/)258.6 ms85.2 ms

Record: runs/2026-10-04_gemma-base/

Confidence

Probabilities that mean what they say

The calibration error is the average gap between the probability an answer states and how often answers like it were right: lower is better, and zero would be exact. Here it is on JevBench's published items, on its standard and hard tiers.

Calibration error on JevBench, per base and tier; lower is betterQwen3.6-35B-A3B: 0.121 on the standard tier, 0.043 on the hard tier; Gemma 4 12B: 0.033 on the standard tier, 0.085 on the hard tier; Gemma 4 31B: 0.035 on the standard tier, 0.091 on the hard tier.0.000.050.100.15Qwen3.6-35B-A3B0.121 standard0.043 hardGemma 4 12B0.033 standard0.085 hardGemma 4 31B0.035 standard0.091 hard

Measured by us on one RTX PRO 6000, vLLM 0.30.0, 2026-10-04, with JevBench's own measure, over ten equal-width bins of the probability; the evaluation card, section 3, has every calibration table and what was fitted on what.

Public boards

The JevBench board

JevBench is an outside board that sends decision questions to each system and scores its answers on intelligence (right answers, on open questions and on sealed ones it never published), calibration, speed and cost, and on a composite of them. Systems are in alphabetical order, and the live board has today's standings.

  • On JevBench v1.6.1 (the board's file, October 2026), Decisio 31B scored 71.69 on the composite (95% interval 69.86 to 72.81) and Jev 1.13.0 scored 71.49 (69.14 to 72.40); the two intervals overlap.
  • Decisio's 95% interval on the composite overlaps those of H2O-Lightning-4B, Quyet-1.0-Large and Jev 1.13.0, so these scores cannot be separated.

The composite, with its 95% interval

JevBench v1.6.1, the board's file, read 2026-10-09

JevBench v1.6.1 composite scores with 95% intervals: ten open-weights systems and Jev 1.13.0Dot and interval chart of JevBench v1.6.1 composite scores with 95% intervals: ten open-weights systems from the board's file and Jev 1.13.0 as a reference row, in alphabetical order; every row's axes are in the table under Axis by axis, below.64666870727476Cygnet68.55 [64.42, 69.94]decider-12b v170.42 [68.09, 71.63]decider-12b v270.88 [68.54, 72.21]Decisio, Gemma 4 12B (ours)68.17 [57.37, 69.65]Decisio, Gemma 4 31B (ours)71.69 [69.86, 72.81]deck-31B68.68 [67.07, 69.74]H2O-Lightning-4B v1.172.52 [69.79, 73.79]Quyet-1.0-Large71.39 [69.84, 72.21]torchcast-decision-12b69.91 [67.22, 71.47]Winnow-12B Q868.89 [66.45, 70.46]Jev 1.13.071.49 [69.14, 72.40]JevBench score, with its 95% interval
JevBench v1.6.1: ten open-weights systems from the board's file, in alphabetical order, with Jev 1.13.0 as a reference row
SystemScore95% interval
Cygnet68.5564.42 to 69.94
decider-12b v170.4268.09 to 71.63
decider-12b v270.8868.54 to 72.21
Decisio, Gemma 4 12B (ours)68.1757.37 to 69.65
Decisio, Gemma 4 31B (ours)71.6969.86 to 72.81
deck-31B68.6867.07 to 69.74
H2O-Lightning-4B v1.172.5269.79 to 73.79
Quyet-1.0-Large71.3969.84 to 72.21
torchcast-decision-12b69.9167.22 to 71.47
Winnow-12B Q868.8966.45 to 70.46
Jev 1.13.071.4969.14 to 72.40
The JevBench board: its file for revision v1.6.1, sha256 5cd8c1332226.

The 12B rows' intervals are wide on the low side. Our temperatures were not fitted on JevBench, but we have read its published items (the evaluation card, section 7.4).

Decisio's rows as the board ran them: Decisio, Gemma 4 12B, run of 2026-10-06 on decisio v0.8.0; Decisio, Gemma 4 12B, run of 2026-10-07 on decisio v0.9.0; Decisio, Gemma 4 31B on decisio v0.8.0.

Axis by axis

ten open-weights systems in alphabetical order and Jev 1.13.0; higher is better on every axis

  • On JevBench v1.6.1, the 31B's sealed-set intelligence was 73.3 and Jev 1.13.0's 61.6; the open-set intelligence was 67.6 and 65.6.
  • The sealed set was never published.
  • On JevBench v1.6.1, Decisio 31B's axes were intelligence 70.4 (Jev 1.13.0 63.6), calibration 88.7 (90.6), speed 91.2 (91.5) and cost 51.7 (54.7).
The board scores each system's intelligence on two sets of items, open ones and sealed ones it keeps back; each row shows both scores, the open one as a ring and the sealed one as a dot.
  1. Decisio, Gemma 4 12B, run of 2026-10-0651.6 sealed53.1 open
  2. Decisio, Gemma 4 12B, run of 2026-10-0751.5 sealed53.1 open
  3. Decisio, Gemma 4 31B73.3 sealed67.6 open
  4. deck-31B75.8 sealed70.2 open
  5. H2O-Lightning-4B59.0 sealed61.1 open
  6. Quyet-1.0-Large71.6 sealed75.2 open
  7. Jev 1.13.061.6 sealed65.6 open
Every number, and what each axis measures
  • Overall: composite score
  • Intelligence: right answers
  • Sealed questions: never published
  • Calibration: probabilities you can trust
  • Speed: faster answers
  • Cost: higher is cheaper
JevBench: ten open-weights systems and Jev 1.13.0 on the composite, the four axes and the sealed questions, with each value in its column
SystemOverallIntelligenceSealed questionsCalibrationSpeedCost
Cygnet68.5554.851.387.091.856.4
decider-12b v170.4260.659.583.890.557.8
decider-12b v270.8863.063.682.090.557.8
Decisio 12B (ours)68.1752.351.689.287.059.3
Decisio 31B (ours)71.6970.473.388.791.251.7
deck-31B68.6873.075.882.286.749.7
H2O-Lightning-4B72.5260.059.090.092.660.3
Quyet-1.0-Large71.3973.471.690.086.950.5
torchcast-decision-12b69.9160.561.182.991.756.4
Winnow-12B Q868.8959.559.283.086.756.6
Jev 1.13.0 (reference)71.4963.661.690.691.554.7

The axes have no published intervals; the composite's is in the chart above.

Source: the board's file for revision v1.6.1; its rows for Decisio ran decisio v0.8.0.

Public boards

The Decision Index board

The Decision Index runs 38 public benchmarks in five areas and scores each system from 0 to 100. Its board lists decisio's rows from our submissions, in its data file of 2026-10-14.

decisio's rows on the board

Decision Index 0.3.2, the board's data file of 2026-10-14

decisio's rows as the Decision Index board lists them, Decision Index 0.3.2, its data file of 2026-10-14
RowPublic indexEdition 0.3 index
Decisio, Gemma 4 31B58.0958.55
Decisio, Gemma 4 12B49.7247.47
Decisio, Qwen3.6-35B-A3B48.4347.72

The rows as the board lists them: Gemma 4 31B on decisio 0.6.0; Gemma 4 12B on decisio 0.4.0; Qwen3.6-35B-A3B on decisio 0.4.0.

The evaluation card, section 8, has our own runs of the whole suite.

Beside other systems

Beside a hosted service and open entries

decisio's three bases beside TypeSafe's Jev 1.13 and three open-weights entries from the Decision Index's public board, as of 2026-10-05. Jev's and the other entries' values come only from the public boards and TypeSafe's public documentation; decisio's are our own runs on the Decision Index 0.2.1.

What you get

decisio from its documentation; Jev 1.13 from TypeSafe's public documentation, read 2026-10-05

The questions a buyer asks, answered for decisio and for Jev 1.13, with a link to each row's sources
QuestiondecisioJev 1.13Sources
Where does my data go?Stays on the machine that runs itTypeSafe's hosted API; not used for trainingSources
What does it cost?No per-call price; you pay for the card$42 per billion input tokens; output freeSources
Does it run on my own GPU, Mac or laptop?Yes: GPU, Mac with MLX, or a laptop with OllamaNo: Hosted API only; its documentation describes no self-hosted optionSources
Longest document?At least 32,768 tokens on a Gemma base, and up to 98,304 on the 31B, as the card holds32k tokens for the state plus the longest question (64k per request)Sources
Options per question?Up to 255Up to 255 on a choiceSources
Many questions in one read?Yes: The state is read once and shared by every questionYes: Every question in a request is evaluated in parallelSources
Images?Yes: Photos in the state; measured on the Qwen base onlyNo: Text onlySources
Probabilities I can trust?Yes: Fitted per base; each base's calibration error is under Confidence, aboveYes: Trained for calibrated decisions, in its documentationSources
Can I teach it with my own examples?Yes: POST /v1/tasks fits calibration and an intent head to your labelled examples, and certifies which answers to trust at the error rate you pick; the model stays frozenNo: Not fine-tuned on customer data; shaped through the requestSources
Same answer every time?Qwen: bit for bit. The Gemma bases: the same top answer on one running server, with the exact numbers able to moveNot stated in its documentation; an alias moves when a release shipsSources
Licence?Apache-2.0A hosted service under TypeSafe's customer agreementsSources
Sources, row by row 11 rows

Each statement about Jev is what TypeSafe's public documentation states, quoted in snapshots/typesafe_docs.json with each page's hash, read 2026-10-05 at 21:44 UTC. Where a row says the documentation does not state something, that is about the pages read, not a claim about the service.

  1. Where does my data go?
    decisio (Where data goes)
    to the machine that runs the server, and nowhere else
    Jev 1.13 (Where data goes)
    "Jev is not trained on customer requests or responses."; zero data retention is offered "for enterprise customers" (models, legal)
  2. What does it cost?
    decisio (Price)
    no per-call price; the card's cost (section 1, e)
    Jev 1.13 (Price)
    "$42 / $0.042" per billion / million tokens, "Charged per input token. Output tokens are free." (models)
  3. Does it run on my own GPU, Mac or laptop?
    decisio (Self-hosting)
    yes: one 80 or 96 GB card on vLLM, the Docker image, or a Mac (below)
    Jev 1.13 (Self-hosting)
    served by "the same endpoint, POST /v1/systemone"; the pages read describe the hosted API and no self-hosted option (models)
    decisio (Mac and Ollama)
    Qwen, Gemma 4 12B and Gemma 4 31B (up to 16,383 tokens of state) on Apple silicon with MLX; the Qwen base on Ollama (aminroudaki/decisio), with Ollama's own prompt and no calibration
    Jev 1.13 (Mac and Ollama)
    the pages read describe the hosted API only
  4. Longest document?
    decisio (Context)
    32,768 tokens per prompt: the state plus one question
    Jev 1.13 (Context)
    "64k tokens per request; 32k tokens for state plus the longest question" (models)
  5. Options per question?
    decisio (Options per question)
    up to 255 named options per choice (more is 422, as there); a score has no cap on its levels
    Jev 1.13 (Options per question)
    "a maximum of 255 options per Choice"; a Score "accepts up to 10" levels (API)
  6. Many questions in one read?
    decisio (Several questions per request)
    the state is read once and shared by every question in the request
    Jev 1.13 (Several questions per request)
    "System One models evaluate every question in a request in parallel." (primitives)
  7. Images?
    decisio (Image input)
    photos in the state, through a second engine on the same card (--image-model); measured on the Qwen base only
    Jev 1.13 (Image input)
    "Text only. String, JSON object, or array of text values. No image, audio, or video input." (models)
  8. Probabilities I can trust?
    decisio (Calibration)
    fitted temperatures per base; JevBench ECE on the standard and hard tiers: Qwen 0.121 and 0.043, Gemma 4 12B 0.033 and 0.085, Gemma 4 31B 0.035 and 0.091
    Jev 1.13 (Calibration)
    "System One models are trained for calibrated decisions: their probabilities are optimized against outcomes to reflect uncertainty." (System One)
  9. Can I teach it with my own examples?
    decisio (Teaching from labelled examples)
    POST /v1/tasks fits a per-task calibration and, for long option lists, an intent head on your labelled examples; the model stays frozen
    Jev 1.13 (Teaching from labelled examples)
    "Jev is not fine-tuned or LoRA-adapted with customer data", and "You shape its answers to your domain through the request rather than through per-account weights" (models)
  10. Same answer every time?
    decisio (Repeatability)
    checkpoints at pinned revisions; Qwen: a repeated request returns the same probabilities bit for bit; Gemma 4 12B: probabilities move between two freshly started servers (up to 0.060 on 200 requests, one exact tie changed) and between a first read and a cached one (up to 0.113), so a repeat is not bit for bit; Gemma 4 31B: a request repeats exactly on a freshly started server; on a server that has already answered it, a repeat can differ (5 of 200 items, up to 0.031, no top answer changed, 31B on vLLM 0.30.0 and 0.31.0); across sessions near-tied answers can move (Evaluation card sections 5 to 7)
    Jev 1.13 (Repeatability)
    "An alias moves when a new release ships, so the answers behind it can change without a change on your side." The pages read do not state whether a repeated request returns the same probabilities (models)
  11. Licence?
    decisio (Licence)
    Apache-2.0; the weights are Qwen's under Apache-2.0 and Google's Gemma 4 under Apache-2.0
    Jev 1.13 (Licence)
    a hosted service under TypeSafe's customer agreements, which "govern your use of TypeSafe" (legal)

The headline

as of 2026-10-05; each cell a value with its 95% interval where there is one

Decision Index 0.2.1, 0 to 100, as of 2026-10-05; decisio's values are our own runsDecider chat · Gemma-4-31B 57.33; Decisio Gemma 4 12B 49.43; Decisio Gemma 4 31B 57.58; Decisio Qwen3.6-35B-A3B 48.06; pplx-decider-v1-27b 56.40; Surogate Rune 26B-A4B v3 57.44; Jev 1.13 57.91.0255075100Decider chat · Gemma-4-31B57.33Decisio Gemma 4 12B49.43Decisio Gemma 4 31B57.58Decisio Qwen3.6-35B-A3B48.06pplx-decider-v1-27b56.40Surogate Rune 26B-A4B v357.44Jev 1.1357.91
Decisio's three bases beside Jev 1.13 and an open entry, Surogate Rune 26B-A4B v3, as of 2026-10-05
MeasureDecisioQwen3.6-35B-A3BDecisioGemma 4 12BDecisioGemma 4 31BJev 1.13Surogate Rune26B-A4B v3
Decision Index 0.2.1 (0 to 100)48.0649.4357.5857.9157.44
Knowledge & Reasoning32.4531.3745.5751.4043.36
Language Understanding51.3951.6661.0262.0263.06
Retrieval & Classification53.5658.0062.7555.4263.52
Tools & Automation68.4371.6275.0475.0971.22
Arts & Human Taste31.5732.5937.4337.6641.90
On JevBench's 231 published questions, correct200200213200not compared (h)
JevBench hard tier, correct82 of 11182 of 11193 of 111not compared (a)not compared (h)
Latency, price or self-hosting cost, context length and what each system is
SystemLatency, median per requestPrice or self-hosting cost, one pass over the 38 index benchmarksContext lengthWhat it is
DecisioQwen3.6-35B-A3B70.5 ms (b)$6.98 (e)32,768 tokens per promptofficial Qwen weights, frozen, letters readout
DecisioGemma 4 12B37.8 ms (b)$6.92 (e)the largest the card holds, 32,768 to 65,536 tokens per promptofficial Gemma weights, frozen, letters readout
DecisioGemma 4 31B56.0 ms (b)$10.28 (e)the largest the card holds, 32,768 to 98,304 tokens per promptofficial Gemma weights, frozen, FP8 on load, letters readout
Jev 1.13524.1 ms (c)$6.05 (f)64k tokens per request; 32k tokens for the state plus the longest question (g)hosted model, TypeSafe's API
Surogate Rune26B-A4B v3120.5 ms (d)$16.53 (e)not on the boardfull fine-tune of google/gemma-4-26B-A4B-it
Decider chatGemma-4-31B108.5 ms (d)$22.18 (e)not on the boardan inference technique on google/gemma-4-31B-it
pplx-decider-v1-27b101.4 ms (d)$24.94 (e)not on the boardfull fine-tune of Qwen/Qwen3.8-27B
The lettered notes 8 notes

(a) JevBench's board reports Jev's hard tier over 220 questions, 109 of them held out and never published, so it is not the same item set as the 111 published ones we can run.

(b) Our server time on one RTX PRO 6000 Blackwell: the Decision Index kit's HTTP wall time to a server on the same machine, one request at a time, over all 150,759 requests.

(c) The board's own measurement of TypeSafe's hosted API: an HTTPS round trip over the internet from its lab, one request at a time, on its latency sample. It includes the network, so it is not the same measurement as (b) and (d).

(d) The board's on-card median on one RTX PRO 6000, one request at a time, on its latency sample.

(e) Self-hosted, estimated as mean request latency × 150,759 requests at $1.48 per card-hour, what a rented card for our runs cost; for the other entries, with the board's mean latency. One request at a time throughout; a server answering requests concurrently would cost less per pass.

(f) The board's recorded API cost of Jev's run over the same benchmarks, at TypeSafe's tariff of $0.042 per million input tokens, output tokens free.

(g) From TypeSafe's documentation (section 3).

(h) Not on JevBench's board.

Every index benchmark

one row per benchmark, by area, as of 2026-10-05

Every benchmark, by area 38 rows

One row per benchmark, grouped by area, with the coverage-adjusted value that feeds the index (raw on the board); ForecastBench shows its Brier loss, where lower is better. n is the number of items behind the interval: the smallest of the benchmark's cases, our scored requests and Jev's cases. Each interval is in runs/2026-10-05_comparison/results.json, and tables.md there is these tables as the script writes them.

These are the values the index counts, so a few differ from the base table above: within 0.001 on the 12B, except GPQA Diamond (two of 196 items).

Knowledge & Reasoning

Knowledge & Reasoning
BenchmarknQwen3.6-35B-A3BGemma 4 12BGemma 4 31BJev 1.13Rune26B-A4B v3Decider chatGemma-4-31Bpplx-decider-v1-27b
GPQA Diamond, accuracy1960.5100.3880.5200.7860.4690.4900.495
GSM8K, accuracy1,3190.5440.5990.8050.7990.7970.8260.674
ChessBench, accuracy5,0000.1620.1650.2200.1720.1870.2230.172
MuSR, accuracy7520.5810.5810.6400.6610.6490.6420.618
SATA-Bench, case exact accuracy1,6500.2320.2640.3230.2640.3480.2930.299
CRUXEval, accuracy5700.6250.6540.8320.7300.7440.7930.749
CLadder, accuracy5,0000.6220.6340.7640.7260.7270.7460.745
HLE, accuracy5010.1140.1060.1500.2040.1060.1480.120
MMLU-Pro, accuracy12,0320.6130.5500.6940.8270.6720.6960.645
BBH, accuracy5,5070.6940.6910.7620.9290.8110.7620.781

Language Understanding

Language Understanding
BenchmarknQwen3.6-35B-A3BGemma 4 12BGemma 4 31BJev 1.13Rune26B-A4B v3Decider chatGemma-4-31Bpplx-decider-v1-27b
ContractNLI, macro-F11230.7330.7430.7260.7170.7700.7360.781
ANLI, macro-F13,2000.6860.7020.7370.7480.7350.7320.707
WinoGrande, accuracy1,2670.8080.7300.8520.9190.8600.8370.852
HellaSwag, accuracy10,0420.9400.8670.9230.9450.9240.9200.940
ACOS, per-review F14000.1740.1550.2150.2950.2680.1880.204
FinEntity, macro-F19790.8030.9050.9310.8700.8840.9090.929
iSarcasmEval, Sarcasm F1 · track A, English1,4000.4740.4270.5060.5050.6040.5110.603
VAST, macro-F13,0060.4900.5910.6870.6460.7660.7320.708
NLI4CT, macro-F15,5000.7720.8130.8350.8410.8040.8290.848
RAGTruth, F1 on hallucinated class2,7000.7020.6960.7830.7650.7680.7690.803

Retrieval & Classification

Retrieval & Classification
BenchmarknQwen3.6-35B-A3BGemma 4 12BGemma 4 31BJev 1.13Rune26B-A4B v3Decider chatGemma-4-31Bpplx-decider-v1-27b
BANKING77, macro-F13,0800.7460.7290.7840.7970.8380.7910.790
CLINC150+OOS, macro-F15,5000.8220.8720.9010.8930.8740.9120.878
BRIGHT, nDCG@102200.4440.4710.4390.4750.4630.4360.487
Amazon ESCI, macro-F15,0000.4500.5350.5400.5520.5530.5310.551
PhishNChips phishing decisions, accuracy2,0000.7520.8130.8750.6250.8090.8750.600
HoVer, accuracy4,0000.7000.6930.7560.7290.8070.7640.742

Tools & Automation

Tools & Automation
BenchmarknQwen3.6-35B-A3BGemma 4 12BGemma 4 31BJev 1.13Rune26B-A4B v3Decider chatGemma-4-31Bpplx-decider-v1-27b
BFCL, case exact accuracy1,6940.9600.9620.9790.9580.9480.9800.976
ToolRet, nDCG@106850.6250.6220.6130.6530.6440.6360.674
API-Bank, accuracy5080.8170.8700.8540.8820.8330.8500.841
Home appliance simulator, case exact accuracy880.3980.4430.6140.5230.4660.6250.739
When2Call, accuracy3,6520.7140.7610.7720.8100.7600.7690.817

Arts & Human Taste

Arts & Human Taste
BenchmarknQwen3.6-35B-A3BGemma 4 12BGemma 4 31BJev 1.13Rune26B-A4B v3Decider chatGemma-4-31Bpplx-decider-v1-27b
BPoMP, accuracy8110.9040.9100.8940.9090.9000.9060.939
Humicroedit, accuracy2,6280.6080.6170.6250.6190.6220.6380.624
POP909-CL, accuracy2,0000.1420.0600.2500.1660.6610.2790.378
cfcolor, accuracy5,0000.6000.6240.6390.6440.6260.6260.644
ForecastBench, Brier loss (lower is better)10,1390.2280.2020.2070.1740.2030.2020.195
Habermas Machine, accuracy1,6760.4470.4650.4840.4590.4220.4790.418
New Yorker, accuracy5280.6880.6330.7390.7010.7410.7390.703

Each value and interval is in runs/2026-10-05_comparison/: runs/2026-10-05_comparison/; Jev's values are the board's, as the evaluation card, section 8.1, reads them. No figure here comes from a call of ours to TypeSafe's API.

The recorded demos, every player

the same games, seeds and checks for every player, on 2026-10-03 and 2026-10-05

The Demos page shows decisio's own runs; here are the other players beside them.

The players 8 rows
The players of the recorded demos, with their models, dates and routes
rowmodel IDdaterouteserving provider
Decisio, Qwen3.6-35B-A3B baseQwen/Qwen3.6-35B-A3B-FP8 @95a723d0 served by decisio v0.6.02026-10-05local server on the card
Decisio, Gemma 4 12B basegoogle/gemma-4-12B-it @707f0a3b served by decisio v0.6.02026-10-05local server on the card
Decisio, Gemma 4 31B basegoogle/gemma-4-31B-it @842da379, FP8 on load, served by decisio v0.6.02026-10-05local server on the card
Cygnetgoogle/gemma-4-12B-it @707f0a3b with the cygnet-recipe decision server @3cf591c6 (T 3.4)2026-10-03local server on the card
Jev 1.13 through its public APIjev-1.13.02026-10-03TypeSafe public APITypeSafe
google/gemini-3.8-flashgoogle/gemini-3.8-flash2026-10-03OpenRouterGoogle AI Studio
anthropic/claude-haiku-4.5anthropic/claude-haiku-4.52026-10-03OpenRouterAnthropic
openai/gpt-6-lunaopenai/gpt-6-luna2026-10-03OpenRouterOpenAI
Browser agent (10 runs of each task) 8 rows
Browser agent (10 runs of each task)
playertravel completedtravel decisions rightreading room completedreading room decisions rightlatency p50 / p95
Decisio, Qwen3.6-35B-A3B base10/1050/7010/1080/140141 / 174
Decisio, Gemma 4 12B base10/1060/6010/1020/20141 / 275
Decisio, Gemma 4 31B base10/1040/5010/1020/20184 / 417
Cygnet10/1060/6010/1020/2058 / 253
Jev 1.13 through its public API10/1060/6010/1020/20130 / 185
google/gemini-3.8-flash10/1060/6010/1020/201308 / 2030
anthropic/claude-haiku-4.510/1040/5010/1020/201248 / 1767
openai/gpt-6-luna10/1060/6810/1020/202878 / 4464
Driving 8 rows
Driving
playerlockstep: drives passedlockstep: motion errors / motion answersrealtime: drives passedrealtime: motion errors / latency effectsdecisions per second (lockstep, realtime)latency p50 / p95 (lockstep)
Decisio, Qwen3.6-35B-A3B base3 of 38 / 5453 of 35 / 27.74, 3.12102 / 158
Decisio, Gemma 4 12B base3 of 325 / 7923 of 36 / 44.01, 2.99218 / 343
Decisio, Gemma 4 31B base3 of 35 / 5683 of 30 / 32.70, 2.37341 / 569
Cygnet3 of 30 / 5743 of 30 / 72.49, 2.30371 / 693
Jev 1.13 through its public API3 of 30 / 7113 of 30 / 26.58, 3.03126 / 178
google/gemini-3.8-flash3 of 30 / 5641 of 31 / 980.59, 0.401319 / 3853
anthropic/claude-haiku-4.53 of 30 / 6093 of 30 / 120.89, 0.961026 / 1574
openai/gpt-6-luna3 of 30 / 8481 of 30 / 1090.30, 0.083195 / 5911
Pong (5 games of up to 45 s, seeds 20260917 to 20260921) 8 rows
Pong (5 games of up to 45 s, seeds 20260917 to 20260921)
playeragrees with the oracle while the ball approachesdecisions on approachdecisions per secondlatency p50 / p95returns per game
Decisio, Qwen3.6-35B-A3B base60% [53, 65]53122.944 / 4722.8 [12.4, 32.8]
Decisio, Gemma 4 12B base62% [56, 66]53922.848 / 5023.2 [21.2, 24.8]
Decisio, Gemma 4 31B base85% [82, 88]167915.068 / 6984.2 [83.0, 86.6]
Cygnet47% [41, 53]45922.941 / 6919.2 [12.8, 26.4]
Jev 1.13 through its public API96% [94, 98]9138.2116 / 16645.8 [45.4, 46.0]
google/gemini-3.8-flash98% [94, 100]950.871131 / 15705.0
anthropic/claude-haiku-4.569% [55, 86]1391.29745 / 10006.8 [6.4, 7.0]
openai/gpt-6-luna98% [95, 100]580.621460 / 27243.4 [3.0, 3.8]

What next

Before you rely on a number

  • Each figure is one card, one run: your card, host and load will move the times.
  • Every time on this page is server time, one request at a time; Scale covers a busy server.
  • Several figures were measured on earlier releases of the same models; each says which.
  • Every decisio figure here was measured on the plain layout, the text then the question; the Gemma models now serve the header layout by default, which answered more accurately on most of what was measured (Choose the prompt layout).

The evaluation cardRequirements and limitsChoosing a model