Leaderboard guide · JevBench v1.4.2 · updated September 26, 2026
JevBench results, explained
JevBench is the independent leaderboard for Jev-style decision models. In JevBench v1.4.2 an open 4B model, decider-4b v2, finished first on the composite score, 0.8 points ahead of Jev. Jev is still the most accurate system in the top ten. Here is the board, what the score measures, and why those two statements are both true.
- 93systems listed89 of them ranked
- 842decisions per system534 public and 308 sealed
- 29.3%chance on the sealed setJev gets 36.7%, the best in the top ten
- 4equally weighted axesIntelligence, calibration, speed, cost
The top ten
Composite first place: decider-4b v2. Highest intelligence: Jev 1.13.0. Best on the hard tier: Cygnet. Fastest: decider-4b v2. No one system leads everything, which is why the composite alone is a poor guide.
| Rank | System | Score | Intelligence | Calibration | Speed | Hard tier | Sealed | Runs as |
|---|---|---|---|---|---|---|---|---|
| #1 | decider-4b v2Mapika | 64.1 | 49.4 | 75.0 | 92.9 | 67.3% | 34.7% | Self-hosted GPU |
| #2 | Jev 1.13.0TypeSafe | 63.3 | 53.1 | 76.3 | 83.3 | 74.1% | 36.7% | Hosted API |
| #3 | JevK5 v0.2.0allebee | 62.0 | 48.9 | 74.5 | 91.1 | 70.0% | 33.1% | Self-hosted GPU |
| #4 | Cygnetblockbrain | 61.8 | 49.5 | 74.9 | 90.7 | 75.5% | 33.8% | Self-hosted GPU |
| #5 | HopperHopitAI | 59.4 | 48.0 | 79.1 | 86.8 | 65.0% | 34.1% | Self-hosted GPU |
| #6 | Winnow-12B Q8EldanRing | 55.6 | 48.3 | 64.8 | 82.3 | 70.9% | 33.1% | Self-hosted GPU |
| #7 | reflex 4Bkshetrajna12 | 54.0 | 47.5 | 70.4 | 68.0 | 63.2% | 28.2% | Self-hosted GPU |
| #8 | djevMaisa | 52.2 | 47.0 | 55.4 | 91.4 | 69.5% | 29.9% | Hosted API |
| #9 | Jev-Omniakhilaaa3 | 51.3 | 46.8 | 64.1 | 81.5 | 75.0% | 32.1% | Self-hosted GPU |
| #10 | metask-jev-4bWayfind | 47.8 | 44.7 | 66.9 | 89.1 | 59.1% | 27.6% | Self-hosted GPU |
The score, intelligence, calibration and speed are 0–100, higher is better. The hard tier is 220 public decisions; sealed is 308 private ones. The fourth axis, cost, is part of the score but not shown; the full table is at the source below.
Same numbers, different weighting
The composite weighs speed and cost as heavily as getting the answer right. JevBench also publishes other views of the same measurements, with the same speed and cost gates. Change the weighting and the top of the board reorders.
| System | Official composite | Accuracy weighted 60% | Intelligence only |
|---|---|---|---|
| decider-4b v2 | #1 | #2 | #5 |
| Jev 1.13.0 | #2 | #1 | #1 |
| JevK5 v0.2.0 | #3 | #4 | #6 |
| Cygnet | #4 | #3 | #4 |
| Hopper | #5 | #5 | #8 |
If your pipeline is high-volume and mostly routine, the composite is a fair guide. If the decisions that cost you money are the ambiguous ones, read the accuracy columns instead.
What changed in JevBench v1.4.2
Published on 25 September 2026, two days after v1.4.0.
- decider-4b v2 from Mapika, an open Qwen3.5-4B rebuild, entered at #1 with 64.1. Jev’s 63.3 did not change; it is now #2.
- Cygnet from blockbrain, frozen Gemma 4 12B with no fine-tuning, entered at #4 with 75.5% on the hard tier, the best in the top ten.
- Jev-Omni, the multimodal Gemma 4 decision model, was added in v1.4.1 and ranks #9.
- Seventeen systems were added over two releases, six in v1.4.1 and eleven in v1.4.2. The list now has 93 systems, 89 of them ranked.
- The scoring formulas and every earlier measurement are unchanged. Systems that were already on the board kept their scores and moved down only because new ones landed above them.
How the score works
Four axes, one harmonic mean, and two penalties that explain most of the surprises on the board.
Intelligence is how often a system picks the right answer, measured above chance, with the sealed decisions counting for 20%. A system that does more than 25 points better on the public items than on the sealed ones loses intelligence in proportion, which is how the benchmark discounts results that may have been tuned to public data.
Calibration asks whether a stated 0.8 really means about 80%. Speed is measured response time, with self-hosted latency doubled plus 0.15s to approximate production load. Cost is the benchmark’s estimate per decision.
The composite is a harmonic mean of the four, so one weak axis drags the whole score down. That is why the most accurate systems measured, such as the thinking modes of djev and OpenJev, rank far below the top ten: they pay heavily on speed and cost.
Compare Jev with the systems on the board
Each page puts Jev and one alternative side by side on the same JevBench numbers, then covers what the benchmark does not measure.
- Jev vs decider-4bThe new first place on JevBench, and what that first place is made of.
- Jev vs JevK5One of three open systems in JevBench’s top four, and where the gap to Jev still is.
- Jev vs CygnetNo training at all, fourth on the board, and ahead of Jev on the hardest public cases.
- Jev vs WinnowOne local model for decisions, chat and images, against a hosted decision model.
- Jev vs djevA faster decision API, against a better calibrated one.
- Jev vs SemIfA fast open rebuild on your own GPU, and where it still falls short.
- Jev vs OpenJevOne name, ten projects. Here is every one, and how the drop-in compares.
- Jev vs LayaA model you fine-tune on your data, against one that works out of the box.
- Jev-OmniThe multimodal Gemma 4 decision model, #9 on the board.
JevBench FAQ
What is JevBench?
An independent benchmark from Benchmark Heaven for Jev-style decision models: systems that take a state and typed questions and return a probability for each option. Its harness is open source. Every system gets the same 534 public and 308 sealed decisions, one request at a time, and is scored on intelligence, calibration, speed and cost.
Which model is first on JevBench?
decider-4b v2 from Mapika, at 64.1 in JevBench v1.4.2. Jev is second at 63.3. decider-4b leads on speed and on the benchmark’s cost estimate; Jev leads on intelligence, calibration, the hard tier and the sealed set.
Why did Jev drop from first to second?
Its score did not change. decider-4b v2 was added on 24 September and scored 0.8 points higher, mainly because a 4B model on a local GPU is faster and cheaper per decision. Weighted toward accuracy, the benchmark’s own view still puts Jev first.
Why are the sealed-set scores so low?
The 308 sealed decisions are private, grouped into hard families such as multi-hop lookups, trade-offs and reworded questions, and no system could be tuned on them. Chance is 29.3%. Jev’s 36.7% is the best in the top ten, and every Jev-style system does far worse there than on the public items.
Is JevBench run by TypeSafe or by Jev AI?
Neither. Benchmark Heaven publishes it independently. Jev AI is an independent service that offers Jev through its own API and is not affiliated with TypeSafe or Benchmark Heaven.
Can I use these other models on Jev AI?
Jev AI runs Jev, plus Laya and Jev-Omni in beta. The other systems on the board are self-hosted or offered by their own authors; the comparison pages link to each project.
Sources
Read on September 26, 2026.
JevBench v1.4.2, published on 25 September 2026, lists 93 systems and ranks 89 of them. Each is scored on the same 534 public decisions (72 easy, 96 standard, 146 judge, 220 hard) plus 308 sealed decisions whose text stays private, one request at a time. The composite is a harmonic mean of four equally weighted axes (intelligence, calibration, speed and cost), so one weak axis pulls it down hard. Sealed decisions make up 20% of intelligence, and a system that does more than 25 points better on public items than on sealed ones loses intelligence in proportion. Every Jev-style system scores far lower on the sealed set than on the public one, Jev included (36.7% against a 29.3% chance level). Version 1.4.2 kept the v1.4.0 formulas and every earlier measurement; it added 17 systems over two releases, which is why ranks moved while scores did not. Self-hosted and demo latency is adjusted by x2 plus 0.15s to approximate production load, so those numbers are not comparable with the raw latency their authors publish.
Every product named on this page belongs to its respective owner and is not affiliated with Jev AI. Figures are quoted from the sources above as they read on September 26, 2026 and can change with the next release of the benchmark.