Leaderboard guide · JevBench v1.6.1 · updated October 9, 2026
JevBench results, explained
JevBench is the independent leaderboard for Jev-style decision models. JevBench v1.6.1 put the hosted APIs on the full decision set and changed how cost is measured. The roster has kept growing inside the release ever since, and after the additions of 8 October Sage 1.3.0 is first at 74.0, H2O-Lightning-4B second at 72.5, Mercury Decide third at 72.4, decisio on Gemma 4 31B fourth at 71.7 and Jev fifth at 71.5, with Quyet-1.0-Large, which led the previous release, sixth at 71.4. On Benchmark Heaven’s headline Capability ranking, which leaves speed and cost out, Jev is fifth. Here is the board, what each ranking measures, and how to read them.
- 173systems on the roster135 of them ranked
- #5Jev on CapabilityIntelligence and calibration averaged, 77.1
- 1,500decisions per system300 open and 1,200 sealed
- 50%of intelligence is sealedDecisions nobody outside the benchmark has seen
The top ten
Composite first place: Sage 1.3.0. Highest intelligence: Quyet-1.0-Large. Best on the sealed half: decisio v0.8.0 on Gemma 4 31B. Fastest: Sage 1.3.0. No one system leads everything, which is why the composite alone is a poor guide.
| Rank | System | Score | Intelligence | Calibration | Speed | Open | Sealed | Runs as |
|---|---|---|---|---|---|---|---|---|
| #1 | Sage 1.3.0Levanto Labs | 74.0 | 65.6 | 91.6 | 93.6 | 65.7 | 65.5 | Hosted API |
| #2 | H2O-Lightning-4BH2O.ai | 72.5 | 60.0 | 90.0 | 92.6 | 61.1 | 59.0 | Self-hosted GPU |
| #3 | Mercury DecideInception | 72.4 | 65.9 | 81.3 | 85.5 | 67.1 | 64.7 | Hosted API |
| #4 | decisio v0.8.0 on Gemma 4 31BAmin Roudaki | 71.7 | 70.4 | 88.7 | 91.2 | 67.6 | 73.3 | Self-hosted GPU |
| #5 | Jev 1.13.0TypeSafe | 71.5 | 63.6 | 90.6 | 91.5 | 65.6 | 61.6 | Hosted API |
| #6 | Quyet-1.0-LargeChinh Nguyen | 71.4 | 73.4 | 90.0 | 86.9 | 75.2 | 71.6 | Self-hosted GPU |
| #7 | decider-12b v2Mapika | 70.9 | 63.0 | 82.0 | 90.5 | 62.4 | 63.6 | Self-hosted GPU |
| #8 | wity-1Wity | 70.8 | 70.3 | 87.9 | 71.8 | 68.3 | 72.3 | Hosted API |
| #9 | decider-12b v1Mapika | 70.4 | 60.6 | 83.8 | 90.5 | 61.8 | 59.5 | Self-hosted GPU |
| #10 | torchcast-decision-12bTorchcast AI | 69.9 | 60.5 | 82.9 | 91.7 | 60.0 | 61.1 | Self-hosted GPU |
The score, intelligence, calibration and speed are 0–100, higher is better. Open and sealed are the two halves of intelligence: 300 open decisions and 1,200 sealed ones, which every system on this board now answers, hosted APIs included. The fourth axis, cost, is part of the score but not shown; the full table is at the source below.
Same numbers, different rules
The official score weighs all four axes equally, with an intelligence floor of 50. JevBench also publishes two alternative orders: B weights intelligence at 40%, and C raises the intelligence floor to 60.
| System | Official (A) | Intelligence 40% (B) | Floor 60 (C) |
|---|---|---|---|
| Sage 1.3.0 | #1 | #1 | #1 |
| H2O-Lightning-4B | #2 | #7 | #2 |
| Mercury Decide | #3 | #4 | #3 |
| decisio v0.8.0 on Gemma 4 31B | #4 | #3 | #4 |
| Jev 1.13.0 | #5 | #6 | #5 |
| Quyet-1.0-Large | #6 | #2 | #6 |
Jev is fifth on the official score and fifth again when the intelligence floor rises to 60, but sixth when intelligence is weighted at 40%, where Quyet-1.0-Large and wity-1 pass it on raw accuracy. Sage 1.3.0 is first under all three rules.
The Capability ranking
Benchmark Heaven leads its page with this view: the average of intelligence and calibration, among the 97 Jev-class systems that fit a frozen budget of at most twice Jev’s v1.5 cost per decision and twice its median latency. It asks how good the answers are, not how cheap they are.
| Capability | System | Score | Official rank |
|---|---|---|---|
| #1 | Quyet-1.0-Large | 81.7 | #6 |
| #2 | decisio v0.8.0 on Gemma 4 31B | 79.6 | #4 |
| #3 | Sage 1.3.0 | 78.6 | #1 |
| #4 | deck-31B | 77.6 | #12 |
| #5 | Jev 1.13.0 | 77.1 | #5 |
| #6 | René-1 31B FP8 | 76.0 | #26 |
| #7 | H2O-Lightning-4B | 75.0 | #2 |
| #8 | SPX-CD Flash | 74.3 | #19 |
| #9 | Mercury Decide | 73.6 | #3 |
| #10 | Bobcat Flash 1.2 | 73.5 | #18 |
The two rankings disagree sharply at the top: decisio on Gemma 4 31B, #4 on the official score, is second here, while deck-31B, only #12 on the official score, is fourth. René-1 31B FP8 is sixth here and #26 there. Mercury Decide, third on the official score, falls to ninth once speed and cost come out, because its calibration is 81.3. Jev-Omni, sixteenth here, has left the top ten of both. wity-1, #8 on the official score, is still absent, because its 1.57s median is slower than the 1.23s latency cap.
What changed in v1.6
A full re-measure on a new pool of decisions, then a new cost rule and sixteen new systems in v1.6.1 — which has kept adding systems since, thirty of them after the release itself.
- v1.6.1 moved the hosted APIs onto the same full 1,500-decision set the self-hosted systems answer, instead of the 600-item subset they were given in v1.6.0, and stopped equating their accuracy axes onto the self-hosted scale. Jev’s own measured numbers shift slightly as a result: intelligence 63.6 rather than 62.4, and 61.6 on the sealed half rather than 63.6.
- v1.6.1 also changed how cost is measured. Every token-priced system is now priced on one common item set — all 1,500 decisions except the 23 that fall outside Jev’s accepted input range — so a system that answered those long items no longer pays for their tokens while one that refused them does not. Refusals still count against intelligence.
- H2O-Lightning-4B v1.1, an H2O.ai fine-tune of Qwen3.5-4B served on stock vLLM, entered on 7 October straight at #2 with 72.5, one place above Jev: a 4B open-weights system matching the hosted APIs on the official score. Its accuracy is lower than Jev’s — intelligence 60.0 against 63.6 — but it is cheaper to run, which the equally weighted cost axis rewards.
- That cost rule reordered the top of the board. Sage 1.3.0, #9 in v1.6.0, is now first at 74.0: its cost axis rose from 43.5 to 58.2 once the long items came out of its bill. Quyet-1.0-Large, the previous leader, fell to #6 at 71.4, with Jev just above it at #5 on 71.5.
- Sixteen open-weights systems were added with the release itself, on the same sealed pool, scorer and cost basis: fourteen from the top 20 of the community Jev Decision Index, then H2O-Lightning-4B v1.1 and Kahn1 4B as add-requests. Three of them are still in the top fifteen — H2O-Lightning-4B at #2, Xor 26B-A4B at #13 and decider chat on Gemma 4 31B at #14 — with Surogate Rune 26B-A4B v3 at #16, GEV-26B-Decide at #17 and Bespoke Nimble 9B v3 at #21 behind them.
- The roster has grown from v1.6.0’s 127 systems to 173, and the ranked order from 92 to 135. Nothing was dropped, so most moves down the board are the new arrivals pushing everything else along: Plumb-4B #10 to #30, decider-4b v2 #17 to #39, JevK5 v0.3 #18 to #41, Imajev-4B #26 to #53 and SemIf #45 to #80.
- v1.6.1 is still taking new systems without a version bump, and the roster moved again between 7 and 8 October: it went from 143 systems and 108 ranked to 165 and 127. No protocol changed and no published score was revised. Two frozen Gemma 4 12B readouts by Amin Roudaki were the first of those additions to land inside the top ten — decisio v0.8.0 at #10 with 68.2 and decisio v0.9.0 at #11 with 67.8 — and they pushed Jev-Omni out of it.
- Later on 8 October the same release took eight more approved systems, again with no version bump, and the roster reached 173 with 135 ranked. Four of the eight landed inside the top ten and moved Jev from #3 to #5: Mercury Decide, Inception’s diffusion decision API, entered at #3 with 72.4; decisio v0.8.0 read out of Gemma 4 31B at #4 with 71.7; and Mapika’s decider-12b v2 and v1 at #7 with 70.9 and #9 with 70.4. Winnow, deck-31B, Cygnet and the Gemma 4 12B decisio row all left the top ten as a result. Mercury Decide and the 31B decisio row are both ahead of Jev under all three of the benchmark’s published orders, so no reweighting puts Jev back above them.
- v1.6 as a whole is a full re-measure, not an addition. Every system with a reproducible recipe was re-run on a new pool of 1,500 decisions: 300 open and 1,200 sealed, against v1.5’s 904 and 720. No v1.6 score can be compared with a v1.5 one.
- The Jev-class cost and latency caps stay frozen at twice Jev’s v1.5 reference, at most $0.0646 per 1,000 decisions and 1.23s, rather than tracking Jev’s own figures — Jev’s median latency fell to 0.24s in v1.6, which would otherwise have shut most of the field out. 97 of the 135 ranked systems now qualify.
- Jev is fifth on the Capability ranking at 77.1, behind Quyet-1.0-Large at 81.7, decisio on Gemma 4 31B at 79.6, Sage 1.3.0 at 78.6 and deck-31B at 77.6. It was fourth before the 8 October additions, third in v1.6.0, and led that ranking in v1.5.6; the 31B decisio row is what moved it down this time, and Sage earlier, because the new cost rule brought Sage inside the Jev-class budget for the first time.
- djev has left the roster, and Vansa-3.4, razorback16’s OpenJev and openjev-sglang were not measured on the new pool. Surogate Rune 26B-A4B v3, absent from v1.6.0, is back and ranked.
How the score works
Four axes, one harmonic mean, and two penalties that explain most of the surprises on the board.
Intelligence is how often a system picks the right answer, measured above chance on 1,500 decisions. Half of it comes from the 1,200 sealed decisions, and a system whose open results beat its sealed results by more than the field median loses intelligence in proportion, which is how the benchmark discounts results that may have been tuned to public data. Choice, yes/no and score requests count equally.
Calibration asks whether a stated 0.8 really means about 80%. Speed is measured response time, with self-hosted latency doubled plus 0.15s to approximate production load. Cost is the benchmark’s estimate per decision.
The composite is a harmonic mean of the four, so one weak axis drags the whole score down, and an axis below 50 scales it down further. That is why some of the most accurate systems measured, such as Eikos-27B at #73 and OpenJev’s thinking mode at #75, rank far below the top ten: they pay heavily on speed and cost.
Compare Jev with the systems on the board
Each page puts Jev and one alternative side by side on the same JevBench numbers, then covers what the benchmark does not measure.
- Jev vs Imajev-4BA 4B model that reads photos, against the model it copies on text.
- Jev vs decider-4bLevel on speed, far behind on the answers.
- Jev vs JevK5A fast open Qwen3.5-4B system, and far behind Jev on the answers.
- Jev vs CygnetNo training at all, thirteenth on the board, and behind Jev on every accuracy axis.
- Jev vs decisioFourth on the board on its biggest base, and it trains nothing of its own.
- Jev vs WinnowOne local model for decisions, chat and images, against a hosted decision model.
- Jev vs djevLevel with Jev when it was last measured, and more expensive to run.
- Jev vs SemIfA fast open rebuild on your own GPU, and where it still falls short.
- Jev vs OpenJevOne name, ten projects. Here is every one, and how the drop-in compares.
- Jev vs LayaA model you fine-tune on your data, against one that works out of the box.
- Jev vs Mercury DecideTwo hosted decision models that take the same request, two places apart.
- Imajev-4B vs Jev-OmniTwo open decision models that read images, on both the text and the image leaderboard.
- Jev-OmniThe multimodal Gemma 4 decision model, #16 on the board and hosted on this site.
JevBench FAQ
What is JevBench?
An independent benchmark from Benchmark Heaven for Jev-style decision models: systems that take a state and typed questions and return a probability for each option. Its harness is open source. Self-hosted systems get the same 300 open and 1,200 sealed decisions, one request at a time, and every system is scored on intelligence, calibration, speed and cost.
Which model is first on JevBench?
It depends on the ranking. On the official score, which weighs intelligence, calibration, speed and cost equally, Sage 1.3.0 is first at 74.0, H2O-Lightning-4B second at 72.5, Mercury Decide third at 72.4, decisio on Gemma 4 31B fourth at 71.7 and Jev fifth at 71.5 in JevBench v1.6.1. On Benchmark Heaven’s headline Capability ranking, which averages intelligence and calibration among Jev-class systems, Quyet-1.0-Large is first at 81.7, decisio on Gemma 4 31B second at 79.6, Sage 1.3.0 third at 78.6, deck-31B fourth at 77.6 and Jev fifth at 77.1.
Where is Imajev-4B, which led v1.4.2.2?
It ranks #53 at 28.7 in v1.6.1, and thirty-fifth on Capability. On the re-measured pool its intelligence is 34.6 against Jev’s 63.6, and it answers yes/no questions barely above chance.
Why did the scores change so much?
v1.6.0 re-measured every system with a reproducible recipe on a new pool of 1,500 decisions, 1,200 of them sealed, under the v1.6 protocol. Smaller 4B rebuilds fell furthest, because the harmonic mean scales a score down once intelligence drops below 50. v1.6.1 then put the hosted APIs on the same full set and priced every token-priced system on one common item set, which moved Sage 1.3.0 from #9 to #1. The scores are not comparable with any v1.5 or v1.4 release.
Is JevBench run by TypeSafe or by Jev AI?
Neither. Benchmark Heaven publishes it independently. Jev AI is an independent service that offers Jev through its own API and is not affiliated with TypeSafe or Benchmark Heaven.
Can I use these other models on Jev AI?
Jev AI runs Jev, plus Laya and Jev-Omni in beta. The other systems on the board are self-hosted or offered by their own authors; the comparison pages link to each project.
Sources
Read on October 9, 2026.
JevBench v1.6 is a full re-measure on a new decision pool, not an update of v1.5. Every system answers 1,500 decisions: 300 open decisions from public sources and 1,200 sealed decisions whose text stays private, one request at a time. v1.6.1 put the hosted APIs on that same full set instead of the 600-item subset v1.6.0 gave them, so their accuracy axes are no longer equated onto the self-hosted scale. The official score is a harmonic mean of four equally weighted axes (intelligence, calibration, speed and cost), so one weak axis pulls it down hard; an axis below chance counts as zero, which zeroes the score. The sealed decisions make up half of intelligence, and a system whose open-minus-sealed gap exceeds the field median loses intelligence in proportion. Choice, noul and score requests carry equal weight. v1.6.1 also prices every token-priced system on one common item set — all 1,500 decisions except the 23 that fall outside Jev’s accepted input range — so a system that answered those long items no longer pays for their tokens while one that refused them does not. Refusals still count against intelligence. Benchmark Heaven also leads its page with a Capability ranking: the mean of intelligence and calibration among Jev-class systems, those inside a budget frozen at twice Jev’s v1.5 cost and median latency. v1.6.1 lists 173 systems and ranks 135; the rest were not re-measured, and 13 of them keep a separately dated v1.5.x score. Self-hosted latency is adjusted by x2 plus 0.15s to approximate production load, so it is not comparable with the raw latency authors publish. Scores are not comparable with any v1.5 or v1.4 release.
Every product named on this page belongs to its respective owner and is not affiliated with Jev AI. Figures are quoted from the sources above as they read on October 9, 2026 and can change with the next release of the benchmark.