JevBench results, explained

What changed since v1.6.1

The board split into open weights and hosted APIs, new systems arrived from fresh sealed draws, method v1.7 counted slow decisions as wrong, and method v1.8 moved self-hosted cost to rented GPU time.

• Method v1.8 changed how Benchmark Heaven prices self-hosted open-weights systems. v1.8.0, on 29 October, replaced each base model’s list price with the GPU time it takes on a rented RTX PRO 6000; intelligence, calibration, latency, API prices and the frozen Jev-class caps did not change, and the board label stays v1.7.48. Most open-weights cost axes rose, so composite scores rose too: 49 of the 235 ranked open-weights systems now score above Jev’s 71.4 reference, against 7 before, and 217 of them now fit the Jev-class caps, against 171. Torchcast Decision 4B v1.3 still leads the composite, now at 80.5, and baselabs mammoth-1 now leads Capability at 85.2. Jev-Omni moved from #30 to #52 on the composite and from #25 to #52 on Capability as other systems passed it. v1.8.1 and v1.8.2, on 30 October, added 13 unranked rows from the 27 October draw. The API board did not change: Jev 1.13.0 is #15 on the composite and #12 on Capability, and Jev 1.14.0 #18. Image JevBench v0.3.0 did not change.
• Board v1.7.48 was revised in place again on 28 and 29 October through releases v1.7.16 to v1.7.23, with no method change. Three self-hosted rows were ranked: KnowLine 4B-Gen5.5 entered the open-weights composite at #49 with 55.6 and the Capability ranking at #49, KnowLine 26B-A4B-Gen2.5 entered at #67 and Lychee-27B at #124, both outside the Capability caps. The open-weights board now ranks 235 systems, 171 of them Jev-class, and every row from #49 down moved down by up to three places; the top ten and the Capability top ten did not change. Jev-Style-2B, Umpire 35B-A3B and VisionJEV 1.0.0 joined the 27 October regular draw, and DecisionNode, DecisionNode Flash and Strike-1 the API board, all shown unranked until that draw is equated. Jev’s places did not move: Jev 1.13.0 is #15 on the API composite and #12 on Capability, and Jev 1.14.0 #18. Image JevBench v0.3.0 did not change.
• Board v1.7.48 was revised in place again on 28 October through releases v1.7.14 and v1.7.15, with no method change and no change to any ranked row. Five self-hosted rows (Torchcast Decision 31B v1.0, Arcbright Elf 1 12B (r2), Jeff-1.1-Qwen-4B, Decisio 12B tuned commit-think and Jeff-1.1-Large) and Microsoft-Decision-1 with updated weights on the API board joined the 27 October regular draw, shown unranked until that draw is equated to the reference scale. Jev’s places did not move: Jev 1.13.0 is #15 on the API composite and #12 on Capability, and Jev 1.14.0 #18. On Image JevBench v0.3.0, a second read of H2O-Lightning-4B v1.4.1 under changed serving settings was added unranked; its ranked row and every image rank stay as they were.
• Board v1.7.48 was revised in place again on 27 October through releases v1.7.11 to v1.7.13, with no method change. v1.7.11 corrected the cost of four Decisio rows to bill the input the engine actually processes, which lowered their cost scores and composites: decisio v0.8.0 on Gemma 4 31B went from #7 with 71.7 to #32 with 73.2, now below Jev’s 71.4 reference, and Decisio 12B tuned commit from #5 with 72.2 to #6 with 71.5, so Chon-1.0-E4B moved up to #5 and Sieve-31B-v2 returned to the top ten at #10. Capability does not depend on cost, so that ranking did not move. v1.7.13 added H2O-Lightning-4B v1.4.1 from a new regular draw, shown unranked until that draw is equated to the reference scale. The API board did not change: Jev 1.13.0 stays #15 on the composite and #12 on Capability, and Jev 1.14.0 #18. Jev-Omni went from #32 to #52. On Image JevBench v0.3.0, H2O-Lightning-4B v1.4.1 entered at #3, moving Imajev-4B to #4 and Jev-Omni to #16.
• Board v1.7.48 was revised in place again on 25 and 26 October through releases v1.7.6 to v1.7.10, with no method change. Eight self-hosted rows joined the open-weights board. Decisio 12B tuned commit entered the composite at #5 with 72.2 and Philly Decider v2 at #10 with 71.2; Jeff-1.1-Medium entered Capability at #4 with 81.9 and the composite at #14. iota-decide-4b-v0.3, GLiClass v4 9B, Decisio v0.9.0 on Qwen3.6-35B-A3B and two D1A rows on Gemma 4 E4B and E2B were also added. Sieve-31B-v2 and Torchcast Decision 4B v1.1 left the composite top ten, and metask-jev-rain-31B-trained the Capability top ten. The open-weights board then ranked 232 systems, 170 of them Jev-class. On the API board, Myles-G 12B is now costed at its public list price, which moved it from #20 to #2, so every API row from #2 to #19 moved down one place without its score changing: Jev 1.13.0 from #14 to #15 on the composite, still #12 on Capability, and Jev 1.14.0 from #17 to #18. Jev-Omni went from #28 to #32.
• Board v1.7.48 was revised in place on 25 October through releases v1.7.2 to v1.7.5, with no method change. Chon-1.0-E4B’s GPU measurement replaced its CPU run, and it entered the open-weights composite at #5 with 71.8; Sieve-31B-v2 entered at #9 with 70.7 and Capability #7; DOPEST 1.0 31B entered Capability at #3 with 82.0 and the composite at #26; Jeetu Decider 2B was also added. Arcbright Elf 1 31B and decider-12b v2 left the composite top ten, and Bobcat 1.3 and deck-31B the Capability top ten. The open-weights board then ranked 224 systems, 163 of them Jev-class, and four loaded-CPU rows still await re-measurement. classify-1 (ScaleDown) joined the API board at #42, which then ranked 48. Jev’s own places did not move: #14 on the API composite and #12 on its Capability ranking. Jev-Omni went from #25 to #28.
• Board v1.7.48 activates method v1.7: a decision over 10 seconds counts as wrong and contributes zero calibration, and Speed uses every decision. Recorded times were re-scored; five loaded-CPU rows await re-measurement. There were then 220 ranked open-weights systems, 47 ranked APIs and one unranked preliminary API result. Jev moved from API #11 to #14 and Capability #9 to #12, with a composite of 71.4. H2O-Lightning-31B now leads open-weights Capability; Arcbright Elf 1 31B enters that ranking at #3 and the composite at #9.
• Release v1.6.42 adds Torchcast Decision 4B v1.3 at #1 on the open-weights composite (76.2) and #11 on Capability (77.5). At that revision the board label stayed v1.7.47, with 197 open-weights systems and 140 inside the Jev-class caps. Carried scores and the method are unchanged; Jev remained API #11 and Capability #9 at that revision.
• Board v1.7.47 was revised in place through releases v1.6.38–v1.6.40 on 22 October: 196 open-weights systems and 44 APIs were ranked, with 139 and 29 in their Jev-class pools. H2O-Lightning-31B enters the open-weights composite at #2 and Capability at #2; iota-decide-4b also joins. Fastino GLiDE, Drex Max and Myles-G 12B join the API board. The method and carried text scores are unchanged.
• At that revision Jev’s composite remained 71.5, but Fastino GLiDE and Drex Max move it from #9 to #11 on the API composite and from #7 to #9 on Capability. H2O-Lightning-4B led the open-weights composite before v1.6.42 and Chon-1.0-12B its Capability ranking. Jev-Omni is #52 on the text composite and #52 on Capability; Imajev-4B is #133.
• Releases v1.6.35–v1.6.37 on 20–21 October re-measured Seb-9B, added Arbiter-4B, Decider-4B, Myles-G 31B and HS1, and corrected the frozen base-model cost estimate for H2O-Lightning-4B and Kahn1. The cost correction affected only their cost axes and composites, returning H2O-Lightning-4B to the composite lead.
• Releases v1.6.30 to v1.6.34, on 19–20 October, revised the v1.7.46 board in place. v1.6.30 applied the common cost basis to token-priced rows of the regular draw, excluding the 15 long items refused by the Jev reference. Intelligence, calibration, speed and Capability stayed unchanged. Torchcast Decision 4B v1.2 then led the composite at 73.0, ahead of H2O-Lightning-4B at 72.5. The cost correction in v1.6.37 reverses those two places.
• Board v1.7.0, on 6 October, split the leaderboard in two. The main board now ranks only open-weights systems that Benchmark Heaven runs on hardware it rents, so cost and speed compare on equal terms; hosted API offerings are ranked on a separate API board, because an API price is a vendor decision that readers cannot reproduce. No score was recomputed: each board is the published order filtered to its own rows.
• Jev 1.13.0 is a hosted API, so it is no longer ranked on the open-weights board. It stays there as a reference row, which also defines the Jev-class cost and latency caps, and it is ranked on the API board, where it is now #15 of 48 on the composite at 71.4 and #12 on that board’s Capability ranking at 77.1.
• At the split, without the hosted APIs, H2O-Lightning-4B leads the open-weights composite at 72.5, with decisio on Gemma 4 31B second at 71.7 and Quyet-1.0-Large third at 71.4; Jev’s reference score of 71.5 falls between the second and the third. Every open-weights rank rose once Sage 1.3.0, Mercury Decide, wity-1 and Jev itself left the order: Winnow moved from #11 to #7, Cygnet from #13 to #10 and Jev-Omni from #16 to #13.
• The API board was rebuilt the same week. Seventeen API offerings were re-measured on a fresh sealed set of 300 items plus the 300 public ones and equated to the full-set scale (v1.7.7), while Jev, Sage 1.3.0 and wity-1 keep their full-set scores. Liquid AI’s d1 answered the full set and entered at #2 with 73.0 (v1.7.8), OpenAI’s Decisions API on gpt-6-luna was ranked from a second fresh set (v1.7.10) and entered at #17, and Microsoft-Decision-1 joined from a native Azure Foundry run (v1.7.31) and entered at #10 at 69.1.
• Board v1.7.42, on 10 October, merged the version tabs into one current board with every system’s latest measurement, and added the systems measured on the fresh sealed draws after v1.6.1; with later additions, 110 open-weights rows across three draws now carry a draw tag. Jeff-1.0-Large, from the fast-lane draw of 9 October, entered the open-weights Capability ranking at #2 with 81.5, a fifth of a point behind Quyet-1.0-Large, and the composite at #9 with 68.6.
• Those draw-tagged rows are shown as published. The anchor systems that would put each new draw on the v1.6.1 scale have not run on it yet, so its offset is zero for now. Benchmark Heaven notes that systems on the new draws score 6 to 13 sealed points below the reference field’s line on the public items everyone shares — either the new draws are harder, or those cohorts were tuned on the public items — so their rows are, if anything, understated until the anchor runs land.
• v1.7.43 and v1.7.44, both on 10 October, are display changes — Pareto charts of Capability against cost and against latency — and changed no score or rank.
• A second regular sealed draw, drawn on 12 October, merged into the board the same day as releases v1.6.10 to v1.6.14. Seven more systems were measured: four paid self-hosted rows, Uniphore-4B-Decider, Torchcast Decision 4B v1.1 and the hosted attest-31B-v1.0. The board label did not change — it still reads v1.7.46 — but the open-weights board grew from 148 ranked rows to 155 and the API board from 27 to 28, so every rank below a new arrival moved. Torchcast Decision 4B v1.1 entered the open-weights top ten at #7 with 69.6 and pushed Cygnet out of it, Jev-Omni slipped from #13 to #14, and the two metask-jev-rain rows entered the Capability ranking at #4 and #5, moving deck-31B to sixth, René-1 31B FP8 to seventh and H2O-Lightning-4B to eighth. attest-31B-v1.0 came in at #7 on the API board. Jev’s own places did not move: fourth on the API composite and second on its Capability ranking.
• v1.7.45 and v1.7.46, both on 11 October, are display changes too. Self-hosted rows now show the measured p50 next to the network-including estimate the ranking still uses, 2× measured plus 150 ms, and that measured value is tagged “on-GPU, no network” to mark it as not directly comparable with the API board’s end-to-end timings. Neither revision changed a score or a rank.
• The second regular draw kept filling after that, through releases v1.6.15 to v1.6.18, and the board label still reads v1.7.46: by 15 October the open-weights board had grown from 155 ranked rows to 162 and the API board from 28 to 30, so every rank below a new arrival moved again. The composite top ten did not change at all. On the Capability ranking a third metask-jev-rain row, the 31B-trained service v3, entered at #3 with 79.6 and pushed decisio on Gemma 4 31B to fourth, the other two metask rows to fifth and sixth, deck-31B to seventh, René-1 31B FP8 to eighth, H2O-Lightning-4B to ninth and Bobcat Flash 1.2 to tenth, with Surogate Rune 26B-A4B v3 falling out of the top ten. The latest arrival, Respan Span-02 at #32 with 51.8, is outside the Jev-class caps, so it moved the composite ranks below it without touching the Capability ranking, whose pool then stood at 117 systems. Jev-Omni held #14 on the composite and slipped from fifteenth to sixteenth on Capability, and Imajev-4B went from #57 to #59. Jev’s own places did not move: fourth on the API composite and second on its Capability ranking.
• The API board grew once more on 15 October without a board-label change. Release v1.6.20 gave SubQ Snap a cost axis from its published tariff, which made it rankable, so the board went from 30 ranked offerings to 31 and SubQ Snap entered at #11; every API rank below it moved down one, and openjev-sglang moved to #24. Its estimated cost is a whisker above the official cap, so it stays outside the cost-capped views and the Jev-class pool stayed at 15. The open-weights board did not move: still 162 ranked rows and the same composite and Capability top tens. No score was recomputed, and Jev’s own places did not move: fourth on the API composite and second on its Capability ranking.
• Releases v1.6.19 to v1.6.27, from 15 to 17 October, kept adding systems under the same v1.7.46 label: the open-weights board grew from 162 ranked rows to 181 and the API board from 31 to 39, with every carried score unchanged. This time Jev’s own places moved. Three hosted offerings entered above it on the API composite — Sqwish’s kettei-fat at #1 with 77.8, Liquid AI’s d1-ultra at #2 with 76.1 and Sqwish’s kettei-chub at #4 with 73.5 — so Jev fell from fourth to #7 without its score changing, and on that board’s Capability ranking kettei-fat, d1-ultra and Bespoke Nimble-v4-max, re-priced inside the cost cap, moved it from second to #5. TypeSafe’s Jev 1.14.0 was measured on 17 October and is #9 on the API board at 70.2, below Jev 1.13.0; the board keeps Jev 1.13.0 as the reference that sets the Jev-class caps. On the open-weights board the composite and Capability top tens did not change, the Jev-class pool grew from 117 to 129, Jev-Omni held #14 and sixteenth on Capability, and Imajev-4B went from #59 to #63.
• Release v1.6.28, also on 17 October, added two more self-hosted systems under the same label, and one of them went straight to the top. Remek Kinas’s basal-omni-1.0-ultra, a Gemma 4 31B with a merged LoRA, measured on the newest sealed draw, entered the open-weights composite at #2 with 71.8 and the headline Capability ranking at #1 with 82.5, the highest intelligence in the composite top ten at 74.1. The open-weights board then ranked 182 systems and the Jev-class pool was 130; the other new row, PurpleMIST-Flash-1.0, is listed without a rank because its base model has no listing decision in the frozen pricing path. Every open-weights rank from #2 down and every Capability place moved down one: decisio on Gemma 4 31B is now #3, Quyet-1.0-Large #4 and second on Capability, Jeff-1.0-Large dropped out of the composite top ten, Bobcat Flash 1.2 out of the Capability top ten, Jev-Omni went to #15 and seventeenth on Capability, and Imajev-4B to #64. Jev’s reference score of 71.5 then fell between the third and fourth open-weights systems. No carried score changed, the API board did not move, and Jev’s own places did not move: #7 on the API composite and #5 on its Capability ranking.
• Release v1.6.29, on 18 October, added two more ranked systems under the same label. Torchcast AI’s Torchcast Decision 4B v1.2, measured on the newest sealed draw, entered the open-weights composite at #2 with 71.8, level with basal-omni-1.0-ultra on the rounded score and above it on speed and cost, and the Capability ranking at #13 with 72.6. Vega-U, a 0.8B reader run on its author’s own CPU server, entered near the bottom and outside the Jev-class caps. The open-weights board then ranked 184 systems and the Jev-class pool was 131. Every open-weights rank from #2 down moved one place, and two places from Vega-U down: decisio on Gemma 4 31B is now #4, Quyet-1.0-Large #5, Jev-Omni #16 and #18 on Capability, Imajev-4B #65, and deck-31B left the composite top ten. Jev’s reference score of 71.5 now falls between the fourth and fifth open-weights systems. The Capability top ten did not change. No carried score changed, the API board did not move, and Jev’s own places did not move: #7 on the API composite and #5 on its Capability ranking.
• The current common cost rule excludes items beyond the Jev reference’s accepted input range separately for each draw: 23 on the original draw, 15 on the regular draw after v1.6.30. Refusals still count as wrong answers. The Jev-class caps stay frozen at twice Jev’s v1.5 reference, at most $0.0646 per 1,000 decisions and 1.23s. 217 of the 235 ranked open-weights systems qualify, and 33 of the 48 API offerings.
• v1.6 as a whole is a full re-measure, not an addition. Every system with a reproducible recipe was re-run on a new pool of 1,500 decisions: 300 open and 1,200 sealed, against v1.5’s 904 and 720. No v1.6 score can be compared with a v1.5 one.
• djev has left the roster, and razorback16’s OpenJev keeps only a dated v1.5 headline. Vansa-3.4 and openjev-sglang, which v1.6.0 did not measure, are now ranked on the API board, at #26 and #38.

How the score works

Four axes, one harmonic mean, and two penalties that explain most of the surprises on the board.

Intelligence is how often a system picks the right answer, measured above chance on 1,500 decisions. Half of it comes from the 1,200 sealed decisions, and a system whose open results beat its sealed results by more than the field median loses intelligence in proportion, which is how the benchmark discounts results that may have been tuned to public data. Choice, yes/no and score requests count equally.

Calibration asks whether a stated 0.8 really means about 80%. Speed is measured response time, with self-hosted latency doubled plus 0.15s to approximate production load. Cost is the benchmark’s estimate per decision: list price for hosted APIs and, since method v1.8.0, rented GPU time for self-hosted open-weights systems.

The composite is a harmonic mean of the four, so one weak axis drags the whole score down, and an axis below 50 scales it down further. That is why one of the most accurate systems measured, OpenJev’s thinking mode, with intelligence 70.2, ranks #72 on the open-weights board, far below the top ten: it pays on speed, and its cost sits level with Jev’s.

JevBench FAQ

Sources

Read on October 31, 2026.

• Benchmark Heaven — JevBench v1.7.48, open-weights board
• Benchmark Heaven — JevBench API board
• fstandhartinger/jevbench — harness and published results

JevBench method v1.7 applies a 10-second limit per decision: a response taking longer than 10 seconds counts as wrong and contributes zero calibration for that item. Speed uses all decisions, including failures. Benchmark Heaven re-scored recorded per-decision times; it did not run every model again. Tokens, draws and raw outputs are unchanged. Since method v1.8.0 (29 October 2026), self-hosted open-weights rows are costed by GPU time on a monthly-rented RTX PRO 6000 with an efficient inference engine instead of base-model list prices; encoders, architectures outside the six calibrated Gemma/Qwen classes and rows without a verified token basis keep their old costs, and API costs and both caps are unchanged. Intelligence, calibration and latency did not move, but most open-weights cost scores rose, so their composites rose and the Jev-class pool grew. Four rows recorded on a loaded CPU host await re-measurement and remain unranked with their dated v1.6 values; five CPU rows were measured again on GPU and replaced in full. The official composite combines intelligence, calibration, speed and cost using the frozen scorer. Each full-set system answers 1,500 decisions, 300 open and 1,200 sealed; most hosted APIs use an equated 600-item set. Half of intelligence comes from sealed decisions. The open-weights and API boards compare the same measurements within separate hosting groups. Board v1.7.48 ranks 235 open-weights systems and 48 API offerings; an additional API result is preliminary and unranked. There are 110 newer ranked open-weights rows tagged with their draw. Jev 1.13.0 remains an unranked reference on the open-weights board and is ranked on the API board. Capability averages intelligence and calibration among systems inside caps frozen at twice Jev’s v1.5 reference: 217 open-weights systems and 33 APIs qualify. The caps do not track current Jev latency. Self-hosted latency uses an estimated 2× measured latency plus 0.15s; API times include the network. Scores and splits on this site use method v1.8 (v1.7 scoring with the v1.8 cost basis); historical v1.6, v1.5 and v1.4 results are not current standings.