Demos

Answers it can stand behind, measured on real tasks

Taught from a few labelled examples, decisio certified which answers could go out with no one checking them. Here are four of those results, then four recorded demos on each of its three models, every decision checked where there was something to check.

Certified answers on real tasks How the demos were judged Run them yourself

Certified answers

Teach it, certify it, automate the sure answers

Four results from the teach-and-certify runs, each on public data and each with its own page section.

43.3%of a support inbox answered on its own at most 5% wrong; 3.2% of those were wrongBANKING77, Gemma 4 31B, decisio 0.14.0
34% to 40%of a coding agent's shell commands decided on their own at most 5% wrong16 rules, Gemma 4 12B on a Mac, decisio 0.14.0
40wrong answers certified before certification paused on drifting trafficCLINC150, Gemma 4 31B, decisio 0.16.0 release candidate
79.9% to 91.6%right over a drifting stream, once the task was taught to say none of these, with the pause onCLINC150 with off-topic traffic, Gemma 4 31B, replayed through the decisio 0.18.0 monitor

See certified answers on real tasks

Recorded runs

Four demos, played as they ran

Each clip is a recorded run, played at the speed it happened. Every base ran in one session on one NVIDIA RTX PRO 6000 Blackwell Workstation Edition (600 W) on 2026-10-05; each card opens on the default base, Gemma 4 31B, and its numbers are that base's.

Pong

Every move is checked against a perfect paddle while the ball approaches.

85%agree with the perfect paddle while the ball approaches (95% interval 82 to 88)
84.2returns per game (95% interval 83.0 to 86.6)
68 msper decision, median (95th percentile 69 ms)

Gemma 4 31B, decisio v0.6.0, 2026-10-05

Every base 3 bases

Pong (5 games of up to 45 s, seeds 20260917 to 20260921)
player	agrees with the oracle while the ball approaches	decisions on approach	decisions per second	latency p50 / p95	returns per game
Decisio, Qwen3.6-35B-A3B base	60% [53, 65]	531	22.9	44 / 47	22.8 [12.4, 32.8]
Decisio, Gemma 4 12B base	62% [56, 66]	539	22.8	48 / 50	23.2 [21.2, 24.8]
Decisio, Gemma 4 31B base	85% [82, 88]	1679	15.0	68 / 69	84.2 [83.0, 86.6]

A game ends when the model's paddle misses five times; the ball advances one step per decision, so returns per game reward fast answers as much as right ones. Gemma 4 31B played all five games to the 45 s limit, which is why it was asked three times as often as the other bases.

Where it was wrong

Pong: decisio's errors, by kind
demo	base	decisions judged	agree	latency	prompt or state	near-tie	judgement	what the errors are
Pong, ball approaching	Qwen3.6-35B-A3B	531	330	0 by construction	12	63	126	mostly "stay" where the perfect paddle moves down (85 of 201); 21 of the misses turned a return into a lost point
	Gemma 4 12B	539	332	0 by construction	4	13	190	mostly "down" where the perfect paddle moves up (97 of 207); 24 turned a return into a lost point
	Gemma 4 31B	1,679	1,421	0 by construction	13	6	239	mostly "stay" where the perfect paddle moves down (210 of 258); across its misses the ball crosses a median 7 units from the paddle, just outside the 6-unit stay band; none cost a point

First frame of the recorded run: Driving, one drive in real time (scenario s1-1): Qwen3.6-35B-A3B base — Driving, one drive in real time (scenario s1-1): Qwen3.6-35B-A3B base

Driving

Every drive-or-stop is checked against the demo's own rules driver on the same snapshot. Lockstep waits for every answer; in realtime the car keeps moving while it waits.

3 of 3drives passed, in lockstep and in realtime
5motion errors in 568 answers, lockstep
341 msper decision in lockstep, median (95th percentile 569 ms)

Gemma 4 31B, decisio v0.6.0, 2026-10-05

Every base 3 bases

Driving
player	lockstep: drives passed	lockstep: motion errors / motion answers	realtime: drives passed	realtime: motion errors / latency effects	decisions per second (lockstep, realtime)	latency p50 / p95 (lockstep)
Decisio, Qwen3.6-35B-A3B base	3 of 3	8 / 545	3 of 3	5 / 2	7.74, 3.12	102 / 158
Decisio, Gemma 4 12B base	3 of 3	25 / 792	3 of 3	6 / 4	4.01, 2.99	218 / 343
Decisio, Gemma 4 31B base	3 of 3	5 / 568	3 of 3	0 / 3	2.70, 2.37	341 / 569

Lockstep waits for every decision, so it measures the decisions alone; realtime keeps the car moving while a request is in flight, as the demo does, so a slow answer arrives too late. Over all of their drives, in lockstep and in realtime:

the Qwen base passed 16 of 18 and 9 of 9 (in lockstep one collision in s2-4 and one drive off the road in s2-5; the rules driver alone passes both);
Gemma 4 12B passed 18 of 18 and 9 of 9;
Gemma 4 31B passed 18 of 18 and 8 of 9 (in realtime a collision in s2-2);

Where it was wrong

Driving: decisio's errors, by kind
demo	base	decisions judged	agree	latency	near-tie	judgement	what the errors are
driving lockstep, motion (18 drives)	Qwen3.6-35B-A3B	4,173	2,934 + 1,207 as worded	0	13	19	30 of 32 a stop where the rules driver goes
	Gemma 4 12B	5,061	3,927 + 1,108 as worded	0	1	25	all 26 a stop where the rules driver goes
	Gemma 4 31B	4,253	3,055 + 1,182 as worded	0	0	16	14 of 16 a stop where the rules driver goes
driving realtime, motion (9 drives)	Qwen3.6-35B-A3B	1,697	1,077 + 607 as worded	7	1	5	all 6 a stop where the rules driver goes
	Gemma 4 12B	2,087	1,471 + 598 as worded	12	0	6	all 6 a stop where the rules driver goes
	Gemma 4 31B	1,554	836 + 702 as worded	12	1	3	all 4 a stop where the rules driver goes

First frame of the recorded run: Browser agent, the travel task: Qwen3.6-35B-A3B base — Browser agent, the travel task: Qwen3.6-35B-A3B base

First frame of the recorded run: Browser agent, the travel task: Gemma 4 12B base — Browser agent, the travel task: Gemma 4 12B base

First frame of the recorded run: Browser agent, the travel task: Gemma 4 31B base — Browser agent, the travel task: Gemma 4 31B base

First frame of the recorded run: Browser agent, the reading room: Qwen3.6-35B-A3B base — Browser agent, the reading room: Qwen3.6-35B-A3B base

Browser agent

Every step is checked against the actions a careful user could take on two fixture tasks.

10 of 10runs completed on each task
91%of 272 labelled steps inside the acceptable set (248)
184 msper decision, median (95th percentile 417 ms)

Gemma 4 31B, decisio v0.6.0, 2026-10-05

Every base 3 bases

Browser agent (10 runs of each task)
player	travel completed	travel decisions right	reading room completed	reading room decisions right	latency p50 / p95
Decisio, Qwen3.6-35B-A3B base	10/10	50/70	10/10	80/140	141 / 174
Decisio, Gemma 4 12B base	10/10	60/60	10/10	20/20	141 / 275
Decisio, Gemma 4 31B base	10/10	40/50	10/10	20/20	184 / 417

On the 272 labelled steps (the operation question alone), the answer is inside the acceptable set on 243 (89%) for the Qwen base, 269 (99%) for Gemma 4 12B and 248 (91%) for Gemma 4 31B.

Where it was wrong

Browser agent: decisio's errors, by kind
demo	base	decisions judged	agree	latency	prompt or state	near-tie	judgement	what the errors are
browser, labelled steps	Qwen3.6-35B-A3B	272	243	0 by construction	5	4	20	DONE before the search is applied; a click on the wrong control
	Gemma 4 12B	272	269	0 by construction	0	1	2
	Gemma 4 31B	272	248	0 by construction	17	0	7	mostly DONE while a typed search is still unsubmitted

In the live browser runs every base completes both tasks every time, but not always by the shortest path. On the travel task the Qwen base re-selects the category once and opens Casa Flora before submitting the typed destination, and Gemma 4 31B opens Casa Flora without submitting it; the fixture's own check does not test the search, so the runs count as completed and the oracle counts those steps as misses.

In the reading room the Qwen base opens the article and then clicks back to the list about six times (its probability for DONE on the article is 0.27 to 0.42) before DONE wins. Gemma 4 12B takes the shortest path in both tasks in every run, and Gemma 4 31B in the reading room.

First frame of the recorded run: Triage, the same tickets before and after registering 200 labelled tickets: Qwen3.6-35B-A3B base — Triage, the same tickets before and after registering 200 labelled tickets: Qwen3.6-35B-A3B base

Triage

Every ticket's queue is checked against its true queue, before and after teaching it 200 labelled tickets.

97.0%of 400 tickets routed right over 20 queues, with no teaching (95% interval 95.2 to 98.5)
97.2%after teaching it 200 labelled tickets: +0.2 points (95% interval -0.5 to +1.2)
69 sto teach it the 200 tickets

Gemma 4 31B, decisio v0.6.0, 2026-10-05

Every base 3 bases

Triage (400 held-out tickets over 20 queues)
base	plain question	after registering 200 labelled tickets	paired difference	macro-F1, plain / registered	latency p50, plain / registered
Qwen3.6-35B-A3B	91.0% [88.2, 93.8]	92.5% [89.8, 95.0]	+1.5 points [+0.2, +3.0]; 7 answers fixed, 1 broken	0.912 / 0.926	42 / 120
Gemma 4 12B	93.8% [91.2, 96.0]	95.2% [93.0, 97.2]	+1.5 points [+0.0, +3.0]; 8 fixed, 2 broken	0.938 / 0.952	50 / 145
Gemma 4 31B	97.0% [95.2, 98.5]	97.2% [95.5, 98.8]	+0.2 points [-0.5, +1.2]; 2 fixed, 1 broken	0.970 / 0.973	71 / 262

Registration was repeated on each base and took 33 s, 35 s and 69 s.

Method

How the demos were judged

Every decision was checked against an oracle, a judge that knows the right answer, with the same seeds, oracles and renderings for every base.

How each demo's decisions are judged
demo	the oracle	what it is not
Pong	a perfect-information paddle: the true ball's crossing of the paddle's plane after wall bounces, and a move toward it, staying inside half a step (6 units)	while the ball moves away there is nothing to judge; those decisions are counted apart
driving	the demo's own rules driver on the same snapshot and the same candidates: the motion (drive or stop) and the manoeuvre	a good driver, not a perfect one; the drive's own score (arrive with no collision, red light, rolled stop sign, failure to yield or a second off the road) is reported beside it
browser agent	the set of actions a careful user could take at each step of two fixture tasks; 272 labelled steps from three places	a label for the operation and the target on two small fixtures
triage	the ticket's true queue	synthetic, templated tickets

How a miss is classed, in this order

Latency effect: the right decision arrived too late (driving only: the answer equals the oracle on the snapshot it was asked about but not when it was applied, or the demo's 1.5 s timeout replaced it).
Prompt or state effect: a written detector per demo, defined before any fix was tested: the model followed the question's own wording where it differs from the oracle (Pong's 5-unit band against the game's 6; the driving state's "at" a light up to 6 m before the line, where the rules driver stops within 1.5 m, counted "as worded" and not as an error), or the state lacked what the decision needs (the browser agent: DONE while a typed search is still unsubmitted).
Near-tie: the chosen option's probability minus the oracle option's is below 0.10.
Judgement error: anything else; reported, not tuned away.

Run them yourself: the demos in decisio's examples, at v0.35.0.

What next

Put it on your own task

Use cases Certify answers Every player of the demos, on Benchmarks