Jev vs decisio

The analysis

Where Jev and decisio really differ, and why the bars above land where they do.

Three systems in the current top ten train nothing at all — deck-31B at #8, Cygnet at #9 and decisio at #10 — and decisio is the newest of the three and by far the most featureful. It is #10 of 127 at 68.2 against Jev’s 71.5 at #3, and unlike the other two it is not a thin shim. It speaks the same /v1/systemone wire format as Jev, accepts up to 255 options per choice exactly as Jev does, prefills a state once and answers many questions against it, and adds things Jev has no equivalent for: registering a recurring question from labelled examples, an opt-in abstention threshold, and images in the state through a second engine.

On the answers Jev is ahead everywhere. Intelligence is 63.6 against 52.3, and 61.6 against 51.6 on the sealed half. By request type Jev leads on choice questions, 77.4 against 69.1, on yes/no, 50.3 against 42.2, and most heavily on score questions, 63.2 against 45.7 — a 17.5-point gap that is the single biggest reason for the difference in the composite.

Where it closes in is the two axes that are not about being right. Calibration is 89.2 against Jev’s 90.6, which is close for a model nobody trained, and its cost score is the better of the two, 59.3 against 54.7. Jev is faster on the benchmark’s measure, 91.5 against 87.0. On Benchmark Heaven’s headline Capability ranking, which averages intelligence and calibration, Jev is fourth at 77.1 and decisio twelfth at 70.7. Jev also stays above it under all three of the benchmark’s orders — third, fourth and third against tenth, thirteenth and fourteenth — so no reweighting closes the gap.

What decisio actually is

decisio, the decision serving layer over frozen open bases, by Amin Roudaki.

decisio is an Apache-2.0 Python package from Amin Roudaki, first published on 30 September 2026. It implements TypeSafe’s published System One wire format and is not affiliated with TypeSafe. Typed questions about a piece of text go in; a probability for every option comes out, each question from one forward pass with nothing generated.

The base model is a choice, not part of the project. Three ship as profiles behind the same routes and features: Qwen3.6-35B-A3B FP8 as the default, google/gemma-4-12B-it in bf16, and google/gemma-4-31B-it quantised to FP8 on load. Each is an official checkpoint at a pinned revision with no adapter and no fine-tuning, and each profile carries its own measured settings. JevBench measured the Gemma 4 12B profile, whose weights take 22.8 GiB in memory and whose fitted temperature is 3.592 for every question type.

Beyond answering, it registers tasks: POST /v1/tasks learns one recurring question from labelled examples, fitting a calibration and, for long option lists, an intent head. Abstention is opt-in per task against a declared “can’t tell” option. Images in the state are served by a second engine on the same card. A state read once is reused from the prefix cache, so a second, different question on the same document answers in about 36ms on the Gemma 4 12B base against about 260ms for the first read.

It runs on one NVIDIA card under vLLM, in Docker, on Apple silicon through MLX, through Ollama on a 16 GB laptop for the smallest Gemma tag, or on CPU as a development stand-in. The software is Apache-2.0; the Gemma weights carry Google’s model terms.

Two decisio versions are ranked, and the newer one is lower

Both v0.8.0 and v0.9.0 of the Gemma 4 12B profile are on the board, at #10 with 68.2 and #11 with 67.8. Their intelligence is the same to a tenth, 52.3 either way: v0.9.0 answers no better and no worse.

The difference is on the other axes. Calibration falls from 89.2 to 87.6 and the speed score from 87.0 to 85.9, and the harmonic mean turns that into the four tenths between them. The change v0.9.0 describes for this base is when a state’s boundary is registered — after the answer rather than before — which the author measured as a faster first read on an idle queue. Whatever it gained there, the benchmark’s adjusted median did not improve.

This page quotes v0.8.0 throughout, because it is the higher-placed of the two and the one the chart shows.

Why its yes/no score is its weakest axis

On yes/no questions decisio scores 42.2 above chance against Jev’s 50.3, its widest relative shortfall after score questions. The underlying behaviour is not random guessing: the benchmark records it as committing to an answer on 81.7% of the yes/no items, and being right on 90.8% of the ones it committed to.

The rest is hedging near the middle of the range, which a chance-corrected score treats as close to no answer at all. decisio has a flag for exactly this, reporting a probability inside the no-answer band at the band’s edge, which changes the calibration rather than the answer; the configuration JevBench measured did not use it. If your code reads the probability rather than just the top label, that distinction is the one to test on your own cases.

Its answers are also almost always well formed: the benchmark records invalid-response rates of 1.7% on choice questions, 1.1% on yes/no and 1.6% on score.

Jev vs decisio FAQ

Other Jev comparisons

Every number on these pages is quoted from a published source and was read on October 8, 2026.

• Jev vs Imajev-4BA 4B model that reads photos, against the model it copies on text.
• Jev vs decider-4bLevel on speed, far behind on the answers.
• Jev vs JevK5A fast open Qwen3.5-4B system, and far behind Jev on the answers.
• Jev vs CygnetNo training at all, ninth on the board, and behind Jev on every accuracy axis.
• Jev vs WinnowOne local model for decisions, chat and images, against a hosted decision model.
• Jev vs djevLevel with Jev when it was last measured, and more expensive to run.
• Jev vs SemIfA fast open rebuild on your own GPU, and where it still falls short.
• Jev vs OpenJevOne name, ten projects. Here is every one, and how the drop-in compares.
• Jev vs LayaA model you fine-tune on your data, against one that works out of the box.
• Jev vs Mercury DecideTwo hosted decision models that take the same request.
• Run Jev locally?The open alternatives in one table, with licences and hardware.
• Jev vs LLMHow a decision model differs from a chat LLM and a BERT classifier.

Sources

• Benchmark Heaven — JevBench v1.6.1
• aminry/decisio — a serving layer for decisions
• google/gemma-4-12B-it on Hugging Face
• TypeSafe documentation (Jev model card)