Decisio

Demos

Pick an error rate, see what it can automate

We taught decisio two public sets of customer intents and a coding agent's command rules from a few labelled examples each, then asked it to certify which answers can go out with no one checking them. Every message, answer and certificate on this page comes from those runs.

Teach your first taskWhat certified meansHow it was run

Inbox autopilot

A support inbox, split by the certificate

These are real messages from BANKING77's test split, answered by Gemma 4 31B after 10 labelled examples per intent. Pick an error rate: the answers the certificate covers go out on their own, and the rest go to a person.

A stricter rate sends more of the inbox to a person. That is what the promise costs.

Error rate

Sent automatically

certified: the answer goes out as it is

  1. When should I receive my virtual card?

    getting_virtual_card0.96right

  2. What is the limit to number of transactions I can do with a disposable card?

    disposable_card_limits0.97right

  3. What do I need to do to verify the source of my funds?

    verify_source_of_funds0.98right

  4. How do I retrieve a trapped card from a ATM?

    card_swallowed0.96right

  5. I want to change my personal details.

    edit_personal_details0.98right

Sent to a person

not certified: a person checks the answer first

  1. I have a 1 euro fee on my statement.

    extra_charge_on_statement0.83right

  2. Where can I find the card PIN?

    get_physical_card0.31right

  3. Can I use American Express to add money into my account?

    supported_cards_and_currencies0.62right

Sent automatically, of all 3,080 test messages
1,335 43.3% of the inbox
Wrong among those
43 3.2% of them
The certificate's threshold
0.94 certified at or above it

BANKING77's test split, credited under How it was run. Gemma 4 31B from its FP8 checkpoint on one RTX PRO 6000 Blackwell, decisio 0.14.0: taught with 770 examples, certified on 1,535 held-out ones. The 48 messages are a seeded random sample of the test split, not picked by hand.

The share automated

How much each rate answers on its own

Each bar is the share of a test split the certificate covered, and how often those answers were wrong. Two intent sets, two bases, 10 labelled examples per intent.

  • 77.6% to 84.2%answered right, untaught and taught: BANKING77 on Gemma 4 31B
  • 73.1% to 82.3%answered right, untaught and taught: BANKING77 on Gemma 4 12B
  • 92.0% to 95.2%answered right, untaught and taught: CLINC150 on Gemma 4 31B
  • 87.3% to 93.1%answered right, untaught and taught: CLINC150 on Gemma 4 12B
The share of each test split certified, by error targetBANKING77, Gemma 4 31B: at 10%, 78.8% certified, 8.1% of them wrong; at 5%, 43.3% certified, 3.2% of them wrong; at 2%, nothing certified. BANKING77, Gemma 4 12B: at 10%, 73.1% certified, 8.5% of them wrong; at 5%, nothing certified; at 2%, nothing certified. CLINC150, Gemma 4 31B: at 10%, 100.0% certified, 4.8% of them wrong; at 5%, 100.0% certified, 4.8% of them wrong; at 2%, 90.9% certified, 2.2% of them wrong. CLINC150, Gemma 4 12B: at 10%, 100.0% certified, 6.8% of them wrong; at 5%, 95.2% certified, 4.5% of them wrong; at 2%, 82.6% certified, 2.4% of them wrong.BANKING77 Gemma 4 31B 10% 78.8%, 8.1% wrong 5% 43.3%, 3.2% wrong 2% nothing certified BANKING77 Gemma 4 12B 10% 73.1%, 8.5% wrong 5% nothing certified 2% nothing certified CLINC150 Gemma 4 31B 10% 100.0%, 4.8% wrong 5% 100.0%, 4.8% wrong 2% 90.9%, 2.2% wrong CLINC150 Gemma 4 12B 10% 100.0%, 6.8% wrong 5% 95.2%, 4.5% wrong 2% 82.6%, 2.4% wrong 0% 50% 100% The share of each test split certified, by error targetBANKING77, Gemma 4 31B: at 10%, 78.8% certified, 8.1% of them wrong; at 5%, 43.3% certified, 3.2% of them wrong; at 2%, nothing certified. BANKING77, Gemma 4 12B: at 10%, 73.1% certified, 8.5% of them wrong; at 5%, nothing certified; at 2%, nothing certified. CLINC150, Gemma 4 31B: at 10%, 100.0% certified, 4.8% of them wrong; at 5%, 100.0% certified, 4.8% of them wrong; at 2%, 90.9% certified, 2.2% of them wrong. CLINC150, Gemma 4 12B: at 10%, 100.0% certified, 6.8% of them wrong; at 5%, 95.2% certified, 4.5% of them wrong; at 2%, 82.6% certified, 2.4% of them wrong.BANKING77Gemma 4 31B 10% 78.8%, 8.1% wrong 5% 43.3%, 3.2% wrong 2% nothing certified BANKING77Gemma 4 12B 10% 73.1%, 8.5% wrong 5% nothing certified 2% nothing certified CLINC150Gemma 4 31B 10% 100.0%, 4.8% wrong 5% 100.0%, 4.8% wrong 2% 90.9%, 2.2% wrong CLINC150Gemma 4 12B 10% 100.0%, 6.8% wrong 5% 95.2%, 4.5% wrong 2% 82.6%, 2.4% wrong

At 10% the certificate covers most of each split. At 2% BANKING77 certifies nothing on either base: when the taught model is most sure, it is still wrong a little more than 2% of the time, and no number of held-out examples changes that. CLINC150's intents barely overlap, so even a strict rate covers most of it.

The promise

200 re-splits, one line to stay under

A certificate promises that, nine times in ten, at most the rate picked of its certified answers are wrong. To watch that hold, each task's held-out and test answers were pooled and split again 200 times: certify on one part, count the mistakes on the other.

Each dot is one split. A dot above the line is a split where the promise missed, and nine in ten is the bar.

The 200 re-splitsBANKING77, Gemma 4 31B: 200 re-splits at a 5% target. The promise held on 191 of them; 9 came in over the target; 16 certified nothing.0% 2% 4% 6% 8% 10% 0% 25% 50% 75% 100% target 5% share of the split certified wrong among the certified The 200 re-splitsBANKING77, Gemma 4 31B: 200 re-splits at a 5% target. The promise held on 191 of them; 9 came in over the target; 16 certified nothing.0% 2% 4% 6% 8% 10% 0% 50% 100% target 5% share of the split certified wrong among the certified
The promise held
191 of 200 splits
Over the rate
9 splits
Certified nothing
16 splits, counted as held

Drift

Traffic it was never taught, and certification pauses

The CLINC150 task was taught its 150 intents and no way to say a message fits none of them. A replay of 5,363 messages then mixed in off-topic ones, from none at message 1,000 to 30 in every 100 by message 4,000, and each one certified was a wrong answer sent out.

The alarm pauses certification

The confidence alarm pauses certification by default: answers keep coming, but none is marked certified until the traffic looks like the examples again.

  • On the CLINC150 stream above, the first paused answer came at message 1,570, with 40 wrong answers certified before it.
  • On a stream whose traffic jumps at message 1,500, the first paused answer came 74 messages after the jump, with 26 wrong answers certified before it. Certification resumed 316 messages after the traffic returned to normal.
  • On normal traffic it paused nothing.

Measured on the decisio 0.16.0 release candidate, Gemma 4 31B FP8, one RTX PRO 6000 Blackwell.

The fix: teach it to say none of these

Give the task one more option, out of scope, taught from 10 off-topic examples, and certify it on held-out examples that include 100 off-topic ones. An off-topic message then has a right answer, and the certificate counts it.

The same drift stream, off-topic rising from 0 at message 1,000 to 30% at 4,000, with the alarm pausing certification, the default.
No out-of-scope optionOut of scope taught
First paused answer1,5702,601
Right, over the whole stream79.9%91.6%
  • Without the option, the confidence alarm pauses the task sooner: at answer 1,570, after 40 wrong certified answers.
  • With the option taught, the answer mix pauses it at answer 2,601, when about 16% of the messages are off-topic. Over the stream, 6.4% of its certified answers are wrong, against 8.3% with an alarm that only reports, and 138 wrong answers go out certified from message 1,001 on, against 394.
  • It paused on all 40 seeds of the monitor's sampling, and never on the same stream without its off-topic messages.

So teach an "other" option and rely on the pause: the option makes the answers right more often, and the pause caps what a shift costs.

Out of scope taught: a replay of recorded readouts through the decisio 0.18.0 monitor, Gemma 4 31B FP8 on one RTX PRO 6000 Blackwell, replayed 2026-10-19; the readouts were recorded on decisio 0.14.0, on the plain layout. No out-of-scope option: the pause measured on the decisio 0.16.0 release candidate.

Gatekeeper

Allow, ask or block, for a coding agent's commands

A coding agent proposes a shell command, and a team has 16 written rules, each saying allow, ask or block. Gemma 4 12B was taught the rules from 20 labelled commands per rule and asked which rule covers each command; the action follows from the rule.

  1. A commandrm -rf build/
  2. Which rule covers it?one question over the 16 rules
  3. The rule's actionallow ask block
  4. Certified?yes: act on it; no: ask a person
  • 34% to 40%of the commands decided on its own at most 5% wrong, with the wrong action in 1.6% to 2.3% of those
  • 70% to 75%decided on its own at most 10% wrong
  • 16 of 1,799certified commands let through when the policy says ask or block, across the four taught versions of the task

That is about 1 in 112. Four were real model mistakes, such as gh stack merge 7; twelve were strict items in the policy's own lists, commands the rule text supports, such as test runs through python -m. All 16 count as errors, because the policy is the ground truth.

When a new kind of command arrives

Database and migration commands, a kind no rule names, rose to 30% of the traffic. None of the 184 was certified, and 160 went to "ask" through the policy's catch-all rule, "any other program".

The lesson beside CLINC150's: a catch-all option is what keeps certification safe when traffic shifts. CLINC150's task had none, so until the alarm paused certification, its off-topic messages went out certified as some other intent.

The 16 rules and their actions
  • inspect allow
  • git_read allow
  • git_local_write allow
  • git_discard ask
  • git_push ask
  • git_push_force_or_protected block
  • test_build_lint allow
  • run_scripts ask
  • install_packages ask
  • edit_workspace allow
  • delete_in_workspace ask
  • outside_workspace block
  • secrets block
  • network_and_cloud ask
  • remote_code_or_privilege block
  • other ask

Gemma 4 12B, the 6-bit MLX conversion mlx-community/gemma-4-12B-it-6bit, decisio 0.14.0, 2026-10-17 and 2026-10-18. Three draws of 20 commands per rule, each read on the same 1,481 test commands; one more version, taught with 10 per rule, was read across two server starts. The sources are under How it was run.

Before you rely on it

What a certificate does not cover

  • Certify where you serve. A certificate holds on the serving setup it was made on: one made on a Mac does not carry to a card, or the reverse.
  • Certify again after upgrading decisio. A certificate covers the release it was made on.
  • Give it a way to say none of these. A certificate holds for messages like its held-out examples, and it cannot cover an answer the task cannot give, such as out of scope.
  • Which examples you teach with matters at 5%. Three random draws of 10 examples per intent on BANKING77 certified very different shares of the test split at 5%; at 10% the draws agree, as the chart below shows.
  • A harder test split moves the rate. Both 2% certificates on CLINC150 came in a little over the rate on its official test split, which is harder than the validation split they were made on: there Gemma 4 12B is right 94.3% of the time and Gemma 4 31B 96.7%. Pooled and re-split, the same answers hold.
  • Some mistakes are labels that overlap. "my wallet was stolen with my cards in it" is labelled compromised_card and was answered lost_or_stolen_card. They count as mistakes here, as they would in a real inbox.
  • What was not tried: more than one machine per base, legal text, long documents, other languages or adversarial messages.

The share certified, three draws of 10 examples per intent, BANKING77

Three teaching draws, by error targetGemma 4 31B at 5%: 43.3%, 10.8%, 54.5%; Gemma 4 31B at 10%: 78.8%, 79.4%, 79.0%; Gemma 4 12B at 5%: 0.0%, 16.8%, 30.7%; Gemma 4 12B at 10%: 73.1%, 71.2%, 70.8%.Gemma 4 31B 10.8% to 54.5% 5% Gemma 4 31B 78.8% to 79.4% 10% Gemma 4 12B 0.0% to 30.7% 5% Gemma 4 12B 70.8% to 73.1% 10% 0% 50% 100% Three teaching draws, by error targetGemma 4 31B at 5%: 43.3%, 10.8%, 54.5%; Gemma 4 31B at 10%: 78.8%, 79.4%, 79.0%; Gemma 4 12B at 5%: 0.0%, 16.8%, 30.7%; Gemma 4 12B at 10%: 73.1%, 71.2%, 70.8%.Gemma 4 31B10.8% to 54.5% 5% Gemma 4 31B78.8% to 79.4% 10% Gemma 4 12B0.0% to 30.7% 5% Gemma 4 12B70.8% to 73.1% 10% 0% 50% 100%

Method

How it was run

Every serving setting at its default. The model's weights never changed: teaching fits a small layer on what the model reads.

Measured on decisio 0.14.0 on 2026-10-16, 2026-10-17 and 2026-10-18; the drift pause on the decisio 0.16.0 release candidate.

The models and the machines
  • Gemma 4 31B, its FP8 checkpoint, on one RTX PRO 6000 Blackwell (96 GB).
  • Gemma 4 12B, the 6-bit MLX conversion, on one 64 GB Apple silicon Mac.

Nothing here is a speed figure.

The intent sets and the commands
  • BANKING77: 77 banking intents. Casanueva, Temcinas, Gerz, Henderson and Vulic, Efficient Intent Detection with Dual Sentence Encoders (2020), PolyAI. CC BY 4.0.
  • CLINC150: 150 intents across 10 domains, with off-topic queries. Larson et al., An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction (EMNLP 2019), Clinc. CC BY 3.0.

The messages on this page are from their test splits, unchanged. The gatekeeper's commands:

  • SWE-Hero OpenHands trajectories: NVIDIA, nvidia/SWE-Hero-openhands-trajectories (shell commands, shard 0). CC BY 4.0.
  • NL2SH-ALFA: westenfelder/NL2SH-ALFA (from nl2bash, tldr-pages, NL2CMD, LinuxCommands and InterCode). MIT (its parts keep their own licences; tldr-pages CC BY 4.0).
  • documentation shell lines: github/docs, kubernetes/website and MicrosoftDocs/azure-devops-docs (shell lines of the documentation). CC BY 4.0.
  • Templated commands and the database commands of the drift replay, generated for this demo by seeded scripts. CC0.
Teaching and the certificate

Each message is one question over every intent, "Which of these labels fits the text?", and the confidence is the answer's top probability. Each task was taught with its examples, and the server fitted a small layer on them.

The certificate was made on held-out examples that share none with the teaching set: drawn from BANKING77's training split, and all of CLINC150's validation split. It finds the lowest confidence at which the held-out mistakes stay under the rate, with a statistical bound (Learn then Test, with Clopper-Pearson bounds).

Certificates are made on the serving path: read again through it, 200 of 200 held-out examples gave the same confidence.

What it promises: nine times in ten over the draw of held-out examples, at most the rate picked of the certified answers is wrong, for traffic like those examples, on the serving setup it was made on.

The re-splits and the drift replay

For each task, the held-out and test answers were pooled and split again 200 times, by intent, into parts of the original sizes. A split that certified nothing counts as held, and the counts show how often that happened.

The drift stream holds every in-scope CLINC150 test message once, plus off-topic messages from its out-of-scope list, read one at a time through the task with its 5% certificate. The monitor was read every 100 messages.

What next

Certify your own task

Certify answersSupport routingCoding-agent gatekeeper