Jev AI

Jev comparison · JevBench v1.7.48 · updated November 14, 2026

Jev vs decisio

Above Jev on Capability on its biggest base, and it trains nothing of its own.

On this site

Jev 1.13.0

TypeSafe · System One model

The open-weights board’s 71.4 reference, and ahead of the 12B profile on intelligence and every request type.

71.4JevBench v1.7.48 · Open-weights reference row · #18 of 54 on the API board · Capability #15

vs

Alternative

decisio

Amin Roudaki · Frozen Gemma 4 12B behind the /v1/systemone wire format

#70 of 261 on Gemma 4 12B, #42 on Gemma 4 31B, and never a weight fine-tuned.

70.6JevBench v1.7.48 · #70 of 261 on the open-weights board · Capability #73

  1. JevHosted API, generally availableWhere it runsdecisioSelf-hosted on vLLM, Docker, Apple silicon or Ollama
  2. JevPost-trained decision modelTrainingdecisioNone — official Gemma 4 12B at a pinned revision, no adapter
  3. JevUp to 255Options per choicedecisioUp to 255
  4. Jev90.6 on JevBenchCalibrationdecisio89.2 on JevBench
  5. Jev63.6 on JevBenchIntelligencedecisio52.3 on JevBench
  6. Jev61.6Sealed decisionsdecisio51.6
  7. Jev63.2 above chanceScore questionsdecisio45.7 above chance
  8. Jev91.2 · 0.24s median over the networkSpeeddecisio86.3 · 0.10s raw on the benchmark’s GPU

Try it on this page

Run Jev 1.14 right here

Edit the example or paste your own text, then run it. Sign in to run: new accounts get 20 welcome credits, and a failed run costs nothing.

Open the full playground

Support ticket triage · Jev 1.14

1 Input

Hi, this is the third time I am writing. Our payouts have failed every day since Monday and my team cannot pay suppliers. I already re-entered the bank details twice. If this is not fixed today we will have to move to another provider.

2 Questions

  • Choice Which team should handle this ticket?
  • Yes / No Does the customer need a response today?
  • Score How frustrated is the customer?
  • Yes / No Does the customer threaten to leave or switch providers?

JevBench v1.7.48, measured the same way

One benchmark measured every system under one method — it ranks 261 open-weights systems on one board and 54 hosted API offerings on another, with the same scores in both — so these bars are comparable with each other in a way that vendor-published figures are not. The full board and how to read it are on the JevBench results page.

Composite score

#3 H2O-Lightning-4B77.9
#29 Xor 26B-A4B74.0
#31 Arcbright Elf 1 31B74.0
#34 Quyet-1.0-Large73.8
API #14 Sage 1.3.073.7
#36 decider-12b v273.7
#42 decisio 31B73.2
#43 decider-12b v173.2
API #17 Mercury Decide72.2
#54 Winnow72.2
#61 deck-31B71.9
#62 Cygnet71.9
API #18 Jev71.4
#68 Jev-Omni71.0
#70 decisio v0.8.070.6
#78 torchcast-12b69.8
API #24 wity-169.0
#112 Plumb-4B50.4
#129 decider-4b43.1
#139 JevK5 v0.339.2
#153 Imajev-4B30.0
#156 Hopper27.7
#183 SemIf16.1
#253 Laya0.0

Plain numbers are ranks on the open-weights board; “API” marks a rank on the separate board for hosted API offerings. Jev, a hosted API, is ranked there, and appears on the open-weights board only as an unranked reference row, where its score falls between #65 and #66. The composite hides where systems actually differ, so the charts below break it apart. Benchmark Heaven’s headline Capability ranking, shown next to each score above, averages intelligence and calibration among Jev-class systems.

Capability, higher is better

IntelligenceHow often it picks the right answer, half from sealed decisions

Jev63.6
decisio52.3

CalibrationWhether 0.8 really means about 80%

Jev90.6
decisio89.2

SpeedMeasured response time

Jev91.2
decisio86.3

Intelligence on open and sealed decisions

Open300 decisions from public sources

Jev65.6
decisio53.1

Sealed1,200 private decisions nobody could tune for

Jev61.6
decisio51.6

By request type, chance-corrected

ChoicePick one of the options

Jev77.4
decisio69.1

NoulProbability that a statement is true

Jev50.3
decisio42.2

ScorePlace the case on a graded scale

Jev63.2
decisio45.7

The sealed half, highlighted, is where results cannot have been tuned: nobody outside the benchmark has seen those decisions. Request types are scored above chance, so 0 means guessing and a value below 0 means worse than guessing.

One thing these charts leave out: the composite score above also weighs a fourth axis, cost. We do not reproduce it here; the full table is at the source below.

Every system on the board

The capability scores are charted above; this is the ranking and the deployment detail behind them. Jev and decisio are highlighted.

ModelRankScoreMeasured latencyRuns on
H2O-Lightning-4B v1.1 (H2O.ai, Qwen3.5-4B)#377.90.03s raw, 0.21s adjustedSelf-hosted GPU
Xor 26B-A4B (Juspay, Gemma 4 26B-A4B)#2974.00.06s raw, 0.27s adjustedSelf-hosted GPU
Arcbright Elf 1 31B#3174.00.06s raw, 0.27s adjustedSelf-hosted GPU
Quyet-1.0-Large (Chinh Nguyen, Gemma 4 31B decoder)#3473.80.12s raw, 0.39s adjustedSelf-hosted, H100
Sage 1.3.0 (Levanto Labs, hosted API)API #1473.70.17s median / 0.41s p95Hosted API
decider-12b v2 (Mapika, Gemma 4 12B)#3673.70.05s raw, 0.26s adjustedSelf-hosted, RTX PRO 6000
decisio v0.8.0 on Gemma 4 31B (Amin Roudaki, frozen base)#4273.20.05s raw, 0.24s adjustedSelf-hosted, H100
decider-12b v1 (Mapika, stock Gemma 4 12B)#4373.20.05s raw, 0.26s adjustedSelf-hosted, RTX PRO 6000
Mercury Decide (Inception, diffusion decision API)API #1772.20.34s median / 0.96s p95Hosted API
Winnow-12B Q8 (EldanRing, Gemma 4 12B)#5472.20.12s raw, 0.40s adjustedSelf-hosted, RTX 6000
deck-31B (krishna765, frozen Gemma 4 31B)#6171.90.13s raw, 0.41s adjustedSelf-hosted GPU
Cygnet (blockbrain, frozen Gemma 4 12B)#6271.90.04s raw, 0.24s adjustedSelf-hosted, RTX PRO 6000
Jev 1.13.0 (TypeSafe)API #1871.40.24s median / 0.31s p95Hosted API
Jev-Omni (akhilaaa3, Gemma 4 12B)#6871.00.16s raw, 0.47s adjustedSelf-hosted, RTX 6000
decisio v0.8.0 on Gemma 4 12B (Amin Roudaki, frozen base)#7070.60.12s raw, 0.38s adjustedSelf-hosted GPU
torchcast-decision-12b (Torchcast AI, Gemma 4 12B)#7869.80.04s raw, 0.24s adjustedSelf-hosted, H100
wity-1 (Wity, closed API, reasoning auto)API #2469.01.85s median / 7.97s p95Hosted API
Plumb-4B (crh225, JevK5 v0.2 + LoRA)#11250.40.02s raw, 0.19s adjustedSelf-hosted, H100
decider-4b v2 (Mapika, Qwen3.5-4B)#12943.10.03s raw, 0.21s adjustedSelf-hosted, RTX 5090
JevK5 v0.3 (allebee, Qwen3.5-4B)#13939.20.02s raw, 0.19s adjustedSelf-hosted, H100
Imajev-4B (Mohit Garg, Qwen3.5-4B LoRA)#15330.00.07s raw, 0.29s adjustedSelf-hosted, RTX 5090
Hopper (HopitAI, Qwen3.5-4B LoRA)#15627.70.06s raw, 0.26s adjustedSelf-hosted, RTX A6000
SemIf (Qwen3.5-4B)#18316.10.06s raw, 0.27s adjustedSelf-hosted, RTX 5090
Laya (ModernBERT-large)#2530.00.93s raw, 2.01s adjustedSelf-hosted, CPU

JevBench method v1.7 applies a 10-second limit per decision: a response taking longer than 10 seconds counts as wrong and contributes zero calibration for that item. Speed uses all decisions, including failures. Benchmark Heaven re-scored recorded per-decision times; it did not run every model again. Tokens, draws and raw outputs are unchanged. Since method v1.8.0 (29 October 2026), self-hosted open-weights rows are costed by GPU time on a monthly-rented RTX PRO 6000 with an efficient inference engine instead of base-model list prices; encoders, architectures outside the six calibrated Gemma/Qwen classes and rows without a verified token basis keep their old costs, and API costs and both caps are unchanged. Intelligence, calibration and latency did not move, but most open-weights cost scores rose, so their composites rose and the Jev-class pool grew. Four rows recorded on a loaded CPU host await re-measurement and remain unranked with their dated v1.6 values; five CPU rows were measured again on GPU and replaced in full. The official composite combines intelligence, calibration, speed and cost using the frozen scorer. Each full-set system answers 1,500 decisions, 300 open and 1,200 sealed; most hosted APIs use an equated 600-item set. Half of intelligence comes from sealed decisions. The open-weights and API boards compare the same measurements within separate hosting groups. Board v1.7.48 ranks 261 open-weights systems and 54 API offerings. There are 136 newer ranked open-weights rows tagged with their draw; since release v1.8.10 (12 November 2026) the 27 October regular draw is equated to the reference scale with four anchor systems (intelligence −2.9, calibration −1.9 points) and 31 of its rows are ranked, while four stay unranked because their top-five places lie within the uncertainty of that shift. Jev 1.13.0 remains an unranked reference on the open-weights board and is ranked on the API board. Capability averages intelligence and calibration among systems inside caps frozen at twice Jev’s v1.5 reference: 243 open-weights systems and 39 APIs qualify. The caps do not track current Jev latency. Self-hosted latency uses an estimated 2× measured latency plus 0.15s; API times include the network. Scores and splits on this site use method v1.8 (v1.7 scoring with the v1.8 cost basis); historical v1.6, v1.5 and v1.4 results are not current standings.

Jev and decisio feature by feature

AttributeJev 1.13.0decisio
What it isHosted System One model from TypeSafe, version jev-1.13.0Apache-2.0 serving layer over a frozen base; measured on google/gemma-4-12B-it, bf16
LicenceProprietary, hostedApache-2.0 software; Gemma 4 weights under Google’s model terms
Request formatThe original /v1/systemoneThe same /v1/systemone, plus /v1/tasks and an abstention route of its own
Question typesNoul, choice, score — many per call, answered in parallelThe same three, many per request against one shared prefill
Options per choiceUp to 255Up to 255
Supported inputsText onlyText, plus images through a second engine on the same card
Learning from your examplesNone — the model is fixedRegisters a task from labelled examples: a calibration, and an intent head for long option lists
Base modelUndisclosed, post-trained by TypeSafeYours to choose: Qwen3.6-35B-A3B, Gemma 4 12B or Gemma 4 31B, each frozen
HardwareNone — it is an API callOne NVIDIA card under vLLM; also Apple silicon, Ollama or CPU for development

What each one is better at

Where Jev wins

  • Intelligence 63.6 against 52.3, and 61.6 against 51.6 on the sealed half.
  • Ahead on all three request types: choice 77.4 against 69.1, yes/no 50.3 against 42.2 and score 63.2 against 45.7.
  • A Capability score of 77.1 against 70.7 for the 12B profile, #73 among open-weights systems, and above it on the composite too.
  • Faster on the benchmark’s measure, 91.2 against 86.3, with nothing to serve or keep running.
  • A released, versioned product rather than a package first published on 30 September 2026.

Where decisio wins

  • #70 of 261 open-weights systems in JevBench v1.7.48 at 70.6 on Gemma 4 12B, and #42 at 73.2 on Gemma 4 31B — above Jev on the composite and #30 on Capability — from bases nobody fine-tuned.
  • Up to 255 options per choice, the same as Jev, and many questions answered against one shared prefill.
  • Calibration 89.2 against Jev’s 90.6, and the better cost score of the two, 67.7 against 54.7.
  • Things Jev has no equivalent for: task registration from labelled examples, opt-in abstention and images in the state.
  • The base is a flag. Three frozen bases are served behind identical routes, and swapping one changes no application code.

The analysis

Where Jev and decisio really differ, and why the bars above land where they do.

decisio trains nothing at all. Its Gemma 4 31B profile is #42 of 261 open-weights systems with 73.2, above Jev’s 71.4 reference score since Benchmark Heaven’s 29 October GPU-time cost basis, and #30 on the Capability ranking, also above Jev. The profile this page charts, Gemma 4 12B, is #70 at 70.6. Either way it is not a thin shim. It speaks the same /v1/systemone wire format as Jev, accepts up to 255 options per choice exactly as Jev does, prefills a state once and answers many questions against it, and adds things Jev has no equivalent for: registering a recurring question from labelled examples, an opt-in abstention threshold, and images in the state through a second engine.

On the answers Jev is ahead everywhere. Intelligence is 63.6 against 52.3, and 61.6 against 51.6 on the sealed half. By request type Jev leads on choice questions, 77.4 against 69.1, on yes/no, 50.3 against 42.2, and most heavily on score questions, 63.2 against 45.7 — a 17.5-point gap that is the single biggest reason for the difference in the composite.

Where it closes in is the two axes that are not about being right. Calibration is 89.2 against Jev’s 90.6, which is close for a model nobody trained, and its cost score is the better of the two, 67.7 against 54.7. Jev is faster on the benchmark’s measure, 91.2 against 86.3. On Benchmark Heaven’s headline Capability ranking, which averages intelligence and calibration, Jev scores 77.1 and the 12B profile is #73 at 70.7 — though the 31B profile is #30 there at 79.6, above Jev. Jev’s reference score sits above the 12B profile on both of the board’s rankings, the composite and Capability, so neither way of reading the board closes that gap.

What decisio actually is

decisio, the decision serving layer over frozen open bases, by Amin Roudaki.

decisio is an Apache-2.0 Python package from Amin Roudaki, first published on 30 September 2026. It implements TypeSafe’s published System One wire format and is not affiliated with TypeSafe. Typed questions about a piece of text go in; a probability for every option comes out, each question from one forward pass with nothing generated.

The base model is a choice, not part of the project. Three ship as profiles behind the same routes and features: Qwen3.6-35B-A3B FP8 as the default, google/gemma-4-12B-it in bf16, and google/gemma-4-31B-it quantised to FP8 on load. Each is an official checkpoint at a pinned revision with no adapter and no fine-tuning, and each profile carries its own measured settings. JevBench measured the Gemma 4 12B profile, whose weights take 22.8 GiB in memory and whose fitted temperature is 3.592 for every question type.

Beyond answering, it registers tasks: POST /v1/tasks learns one recurring question from labelled examples, fitting a calibration and, for long option lists, an intent head. Abstention is opt-in per task against a declared “can’t tell” option. Images in the state are served by a second engine on the same card. A state read once is reused from the prefix cache, so a second, different question on the same document answers in about 36ms on the Gemma 4 12B base against about 260ms for the first read.

It runs on one NVIDIA card under vLLM, in Docker, on Apple silicon through MLX, through Ollama on a 16 GB laptop for the smallest Gemma tag, or on CPU as a development stand-in. The software is Apache-2.0; the Gemma weights carry Google’s model terms.

Three decisio rows are ranked, and the newer 12B version is lower

Both v0.8.0 and v0.9.0 of the Gemma 4 12B profile are on the board, at #70 with 70.6 and #75 with 70.1. Their intelligence is the same to a tenth, 52.3 either way: v0.9.0 answers no better and no worse. Since 8 October v0.8.0 on Gemma 4 31B is ranked too, now at #42 with 73.2 after a 27 October cost correction.

The difference is on the other axes. Calibration falls from 89.2 to 87.6 and the speed score from 86.3 to 85.3, and the harmonic mean turns that into the five tenths between them. The change v0.9.0 describes for this base is when a state’s boundary is registered — after the answer rather than before — which the author measured as a faster first read on an idle queue. Whatever it gained there, the benchmark’s adjusted median did not improve.

This page quotes v0.8.0 on Gemma 4 12B throughout, because it is the profile the benchmark measured first and the one the chart shows; the 31B row is called out wherever it changes the answer.

Why its yes/no score is its weakest axis

On yes/no questions decisio scores 42.2 above chance against Jev’s 50.3, its widest relative shortfall after score questions. The underlying behaviour is not random guessing: the benchmark records it as committing to an answer on 81.7% of the yes/no items, and being right on 90.8% of the ones it committed to.

The rest is hedging near the middle of the range, which a chance-corrected score treats as close to no answer at all. decisio has a flag for exactly this, reporting a probability inside the no-answer band at the band’s edge, which changes the calibration rather than the answer; the configuration JevBench measured did not use it. If your code reads the probability rather than just the top label, that distinction is the one to test on your own cases.

Its answers are also almost always well formed: the benchmark records invalid-response rates of 1.7% on choice questions, 1.1% on yes/no and 1.6% on score.

So which should you use?

Pick Jev if

  • Accuracy is what you are buying: Jev leads on intelligence, the sealed half and all three request types.
  • Score questions matter — the gap there is the widest of any axis, 63.2 against 45.7.
  • You would rather call an API than operate a GPU, a server and a base-model choice.
  • You want one versioned model with published limits, not a configuration you have to reproduce.

Pick decisio if

  • You want to keep the decision layer and the weights under your own control, on your own card, under Apache-2.0.
  • Your questions recur and you have labelled examples: registering a task is a lever Jev does not offer.
  • You need images in the state, or you want the option of moving to a stronger base without touching your code.
  • Volume is high and routine, and a better cost score matters more than the accuracy gap.

Try Jev on the cases you are actually arguing about

decisio is a strong argument that the serving layer, not the training run, is what most decision work is missing — behind Jev on the composite on the 12B base, and ahead of it on both the composite and Capability on the 31B one. Both are worth measuring on your own decisions rather than on either ranking. Run them on Jev here with no setup and keep the answers.

Jev vs decisio FAQ

Is decisio better than Jev?

It depends which base. On Gemma 4 12B, the profile this page charts, JevBench v1.7.48 ranks decisio #70 of 261 open-weights systems at 70.6, against 71.4 for Jev, shown there as an unranked reference, and Jev leads on intelligence, 63.6 against 52.3, on the sealed half, 61.6 against 51.6, and on all three request types. On Gemma 4 31B decisio is #42 at 73.2, 1.8 points above Jev on the composite, and ahead on intelligence, 70.4 against 63.6, and on Capability, 79.6 against 77.1, while Jev keeps the better calibration, 90.6 against 88.7.

Is decisio fine-tuned?

No. Every base it serves is an official checkpoint at a pinned revision with no adapter and no fine-tuning. The fitted values are calibration temperatures, 3.592 for every question type on the Gemma 4 12B profile JevBench measured.

Which base did JevBench measure?

google/gemma-4-12B-it in bf16 is the profile that places #70 and the one this page charts. The Gemma 4 31B profile was added to the board on 8 October and is now #42 on the composite and #23 on Capability, the highest of the three decisio rows on both, which matches the author’s own view that it is the strongest base. decisio’s default base, Qwen3.6-35B-A3B, is still not ranked.

What hardware does decisio need?

JevBench ran the Gemma 4 12B profile on one RTX 6000 Ada with 48 GB. That base takes 22.8 GiB in memory and does not fit a 32 GB card under vLLM. It also runs on Apple silicon through MLX, through Ollama on a 16 GB laptop for the smallest Gemma tag, and on CPU as a development stand-in.

Can I point the TypeSafe SDK at decisio?

It implements TypeSafe’s published System One wire format, so requests in that shape are what it accepts. It is a third-party project and not affiliated with TypeSafe, and it adds routes of its own for task registration and abstention.

Why are there three decisio rows on the board?

Two versions of the Gemma 4 12B profile were measured, v0.8.0 at #70 with 70.6 and v0.9.0 at #75 with 70.1 — their intelligence is identical at 52.3, and v0.9.0 scores lower on calibration and speed. v0.8.0 on Gemma 4 31B is the third row, added on 8 October and now #42 with 73.2. This page quotes the 12B v0.8.0 row and names the 31B one where it matters.

Other Jev comparisons

Every number on these pages is quoted from a published source and was read on November 14, 2026.

Sources

decisio and every other product named on this page belongs to its respective owner and is not affiliated with Jev AI. Figures are quoted from the sources above as they read on November 14, 2026 and can change.