Jev AIJev comparison · JevBench v1.7.48 · updated October 31, 2026
No training at all, forty-sixth on the open-weights board, and behind Jev on every accuracy axis.
On this site
TypeSafe · System One model
A 71.4 reference score within a point of Cygnet’s, and ahead on intelligence, the sealed half and calibration.
71.4JevBench v1.7.48 · Open-weights reference row · #15 of 48 on the API board · Capability #12
vs
Alternative
blockbrain · Frozen Gemma 4 12B with a one-token readout
#46 of 235 open-weights systems with no training at all, and #56 on Capability.
71.9JevBench v1.7.48 · #46 of 235 on the open-weights board · Capability #56
Try it on this page
Edit the example or paste your own text, then run it. Sign in to run: new accounts get 20 welcome credits, and a failed run costs nothing.
Support ticket triage · Jev 1.14
1 Input
Hi, this is the third time I am writing. Our payouts have failed every day since Monday and my team cannot pay suppliers. I already re-entered the bank details twice. If this is not fixed today we will have to move to another provider.
2 Questions
One benchmark measured every system under one method — it ranks 235 open-weights systems on one board and 48 hosted API offerings on another, with the same scores in both — so these bars are comparable with each other in a way that vendor-published figures are not. The full board and how to read it are on the JevBench results page.
Plain numbers are ranks on the open-weights board; “API” marks a rank on the separate board for hosted API offerings. Jev, a hosted API, is ranked there, and appears on the open-weights board only as an unranked reference row, where its score falls between #49 and #50. The composite hides where systems actually differ, so the charts below break it apart. Benchmark Heaven’s headline Capability ranking, shown next to each score above, averages intelligence and calibration among Jev-class systems.
IntelligenceHow often it picks the right answer, half from sealed decisions
CalibrationWhether 0.8 really means about 80%
SpeedMeasured response time
Open300 decisions from public sources
Sealed1,200 private decisions nobody could tune for
ChoicePick one of the options
NoulProbability that a statement is true
ScorePlace the case on a graded scale
The sealed half, highlighted, is where results cannot have been tuned: nobody outside the benchmark has seen those decisions. Request types are scored above chance, so 0 means guessing and a value below 0 means worse than guessing.
One thing these charts leave out: the composite score above also weighs a fourth axis, cost. We do not reproduce it here; the full table is at the source below.
The capability scores are charted above; this is the ranking and the deployment detail behind them. Jev and Cygnet are highlighted.
| Model | Rank | Score | Measured latency | Runs on |
|---|---|---|---|---|
| H2O-Lightning-4B v1.1 (H2O.ai, Qwen3.5-4B) | #2 | 77.9 | 0.03s raw, 0.21s adjusted | Self-hosted GPU |
| Xor 26B-A4B (Juspay, Gemma 4 26B-A4B) | #21 | 74.0 | 0.06s raw, 0.27s adjusted | Self-hosted GPU |
| Arcbright Elf 1 31B | #23 | 74.0 | 0.06s raw, 0.27s adjusted | Self-hosted GPU |
| Quyet-1.0-Large (Chinh Nguyen, Gemma 4 31B decoder) | #25 | 73.8 | 0.12s raw, 0.39s adjusted | Self-hosted, H100 |
| Sage 1.3.0 (Levanto Labs, hosted API) | API #11 | 73.7 | 0.17s median / 0.41s p95 | Hosted API |
| decider-12b v2 (Mapika, Gemma 4 12B) | #27 | 73.7 | 0.05s raw, 0.26s adjusted | Self-hosted, RTX PRO 6000 |
| decisio v0.8.0 on Gemma 4 31B (Amin Roudaki, frozen base) | #32 | 73.2 | 0.05s raw, 0.24s adjusted | Self-hosted, H100 |
| decider-12b v1 (Mapika, stock Gemma 4 12B) | #33 | 73.2 | 0.05s raw, 0.26s adjusted | Self-hosted, RTX PRO 6000 |
| Mercury Decide (Inception, diffusion decision API) | API #14 | 72.2 | 0.34s median / 0.96s p95 | Hosted API |
| Winnow-12B Q8 (EldanRing, Gemma 4 12B) | #39 | 72.2 | 0.12s raw, 0.40s adjusted | Self-hosted, RTX 6000 |
| deck-31B (krishna765, frozen Gemma 4 31B) | #45 | 71.9 | 0.13s raw, 0.41s adjusted | Self-hosted GPU |
| Cygnet (blockbrain, frozen Gemma 4 12B) | #46 | 71.9 | 0.04s raw, 0.24s adjusted | Self-hosted, RTX PRO 6000 |
| Jev 1.13.0 (TypeSafe) | API #15 | 71.4 | 0.24s median / 0.31s p95 | Hosted API |
| Jev-Omni (akhilaaa3, Gemma 4 12B) | #52 | 71.0 | 0.16s raw, 0.47s adjusted | Self-hosted, RTX 6000 |
| decisio v0.8.0 on Gemma 4 12B (Amin Roudaki, frozen base) | #54 | 70.6 | 0.12s raw, 0.38s adjusted | Self-hosted GPU |
| torchcast-decision-12b (Torchcast AI, Gemma 4 12B) | #62 | 69.8 | 0.04s raw, 0.24s adjusted | Self-hosted, H100 |
| wity-1 (Wity, closed API, reasoning auto) | API #20 | 69.0 | 1.85s median / 7.97s p95 | Hosted API |
| Plumb-4B (crh225, JevK5 v0.2 + LoRA) | #96 | 50.4 | 0.02s raw, 0.19s adjusted | Self-hosted, H100 |
| decider-4b v2 (Mapika, Qwen3.5-4B) | #111 | 43.1 | 0.03s raw, 0.21s adjusted | Self-hosted, RTX 5090 |
| JevK5 v0.3 (allebee, Qwen3.5-4B) | #119 | 39.2 | 0.02s raw, 0.19s adjusted | Self-hosted, H100 |
| Imajev-4B (Mohit Garg, Qwen3.5-4B LoRA) | #133 | 30.0 | 0.07s raw, 0.29s adjusted | Self-hosted, RTX 5090 |
| Hopper (HopitAI, Qwen3.5-4B LoRA) | #136 | 27.7 | 0.06s raw, 0.26s adjusted | Self-hosted, RTX A6000 |
| SemIf (Qwen3.5-4B) | #160 | 16.1 | 0.06s raw, 0.27s adjusted | Self-hosted, RTX 5090 |
| Laya (ModernBERT-large) | #227 | 0.0 | 0.93s raw, 2.01s adjusted | Self-hosted, CPU |
JevBench method v1.7 applies a 10-second limit per decision: a response taking longer than 10 seconds counts as wrong and contributes zero calibration for that item. Speed uses all decisions, including failures. Benchmark Heaven re-scored recorded per-decision times; it did not run every model again. Tokens, draws and raw outputs are unchanged. Since method v1.8.0 (29 October 2026), self-hosted open-weights rows are costed by GPU time on a monthly-rented RTX PRO 6000 with an efficient inference engine instead of base-model list prices; encoders, architectures outside the six calibrated Gemma/Qwen classes and rows without a verified token basis keep their old costs, and API costs and both caps are unchanged. Intelligence, calibration and latency did not move, but most open-weights cost scores rose, so their composites rose and the Jev-class pool grew. Four rows recorded on a loaded CPU host await re-measurement and remain unranked with their dated v1.6 values; five CPU rows were measured again on GPU and replaced in full. The official composite combines intelligence, calibration, speed and cost using the frozen scorer. Each full-set system answers 1,500 decisions, 300 open and 1,200 sealed; most hosted APIs use an equated 600-item set. Half of intelligence comes from sealed decisions. The open-weights and API boards compare the same measurements within separate hosting groups. Board v1.7.48 ranks 235 open-weights systems and 48 API offerings; an additional API result is preliminary and unranked. There are 110 newer ranked open-weights rows tagged with their draw. Jev 1.13.0 remains an unranked reference on the open-weights board and is ranked on the API board. Capability averages intelligence and calibration among systems inside caps frozen at twice Jev’s v1.5 reference: 217 open-weights systems and 33 APIs qualify. The caps do not track current Jev latency. Self-hosted latency uses an estimated 2× measured latency plus 0.15s; API times include the network. Scores and splits on this site use method v1.8 (v1.7 scoring with the v1.8 cost basis); historical v1.6, v1.5 and v1.4 results are not current standings.
| Attribute | Jev 1.13.0 | Cygnet |
|---|---|---|
| What it is | Hosted System One model from TypeSafe, version jev-1.13.0 | Frozen google/gemma-4-12B-it on stock vLLM 0.30.0, plus an MIT shim |
| Licence | Proprietary, hosted | MIT shim; Gemma 4 weights under Apache-2.0 and Google’s prohibited-use policy |
| Request format | POST /v1/systemone | The shim accepts /v1/systemone requests and makes one chat call per decision |
| Question types | Noul, choice, score — many per call, answered in parallel | Noul, choice, score — one request per decision |
| Options per choice | Up to 255 | Up to 26; more is rejected with HTTP 422 |
| Context | 64k tokens per request | 16,384 tokens; longer inputs are rejected |
| Calibration | Post-trained with RLCD; JevBench 90.6 | One temperature, 3.4, fitted on the author’s own items; JevBench 87.0 |
| Hardware | None — it is an API call | A 48 GB GPU in the author’s runs; other cards not measured |
Where Jev and Cygnet really differ, and why the bars above land where they do.
Cygnet is still the clearest evidence that a general model, read the right way, can do much of what a trained decision model does. It led v1.5.6; on the re-measured v1.6 pool it is #46 of 235 open-weights systems at 71.9, against 71.4 for Jev, which the board shows as an unranked reference. What puts it ahead on the composite is the cheap side of the board: speed 91.4 against Jev’s 91.2, from a 37ms raw median on the benchmark’s GPU, and a clearly better cost estimate since Benchmark Heaven’s 29 October GPU-time cost basis, 66.8 against 54.7.
On the answers Jev is now ahead across the board: 63.6 against 54.8 on intelligence, and 61.6 against 51.3 on the sealed half. The split by request type that used to favour Cygnet on yes/no questions has closed — Jev leads there too, though only by 50.3 against 48.7 above chance, and more clearly on choice questions, 77.4 against 69.1, and score questions, 63.2 against 46.8. Calibration is 90.6 against 87.0, and on Benchmark Heaven’s headline Capability ranking, which averages intelligence and calibration, Jev scores 77.1 and Cygnet is #56 at 70.9.
The practical limits are the bigger difference. Cygnet rejects inputs over 16,384 tokens and choices with more than 26 options, and sends one chat request per decision. It has been measured on three GPU models, and the repository appeared on 24 September: it is a recipe you run, not a service.
Cygnet, typed decisions read from frozen Gemma 4 12B, by blockbrain.
Cygnet is a recipe from blockbrain. It runs google/gemma-4-12B-it at a pinned revision on unmodified vLLM 0.30.0, with a small shim in front, written with the Python standard library, that accepts JevBench’s /v1/systemone requests. Nothing about the model is trained or merged.
For each decision the shim sends one chat request with the state, the instructions and the options relabelled A, B, C and so on, and asks for one letter. vLLM restricts the answer position to those letters and returns their log-probabilities; the shim sums every token that decodes to each letter, renormalises over the options and applies a calibration temperature of 3.4. Output is one token per decision. The author credits the one-token readout to NInfer.
The temperature was fitted on 241 items the author generated; the README states that JevBench’s public items were used only to measure, never to fit. On the public set the author measured 203 of 231 correct on both an RTX A6000 and an L40S, with a 50–66ms median on the standard tier.
The shim is MIT. The weights are Google’s Gemma 4 12B under Apache-2.0, subject to Google’s Gemma Prohibited Use Policy.
Cygnet gets remarkably far with no training, which makes it worth measuring on your own cases rather than taking either ranking on trust. Run them on Jev here with no setup and keep the answers; the gap on your decisions is the only one that matters.
Not on the answers. In JevBench v1.7.48 Cygnet is #46 of 235 open-weights systems at 71.9, and Jev, shown there as an unranked reference, scores 71.4 — Cygnet’s half-point edge on the composite comes from its cost score. Jev leads on intelligence, 63.6 against 54.8, on the sealed half, 61.6 against 51.3, and on all three request types. Cygnet led the release before v1.6, v1.5.6, on the old pool; the full re-measure in v1.6 reversed that.
No. It runs Google’s gemma-4-12B-it unchanged on stock vLLM. The only fitted value is one calibration temperature, 3.4, which the author fitted on 241 self-generated items rather than on JevBench’s public set.
The author measured it on an RTX A6000 and an L40S, both 48 GB, and JevBench measured it on an RTX PRO 6000. Other cards have not been measured.
Neither. It is a third-party recipe from blockbrain. Winnow also uses Gemma 4 12B but fine-tunes it; Cygnet uses the stock model.
Every number on these pages is quoted from a published source and was read on October 31, 2026.
Cygnet and every other product named on this page belongs to its respective owner and is not affiliated with Jev AI. Figures are quoted from the sources above as they read on October 31, 2026 and can change.