On this site
Jev 1.13.0
TypeSafe · System One model
Ahead overall, on the sealed set and on calibration, with far fewer limits.
63.3JevBench v1.4.2 rank #2
Jev comparison · JevBench v1.4.2 · updated September 26, 2026
Jev vs Cygnet
No training at all, fourth on the board, and ahead of Jev on the hardest public cases.
vs
Alternative
Cygnet
blockbrain · Frozen Gemma 4 12B with a one-token readout
Fourth of 89 with no training, and the best hard-tier score in the top ten.
61.8JevBench v1.4.2 rank #4
- JevHosted API, generally availableWhere it runsCygnetSelf-hosted on vLLM; measured on 48 GB and 96 GB GPUs
- JevPost-trained decision modelTrainingCygnetNone — stock Gemma 4 12B plus one temperature
- Jev94.5% correctEvaluation-style casesCygnet95.2% correct
- Jev74.1% correctHardest test casesCygnet75.5% correct
- Jev36.7% correctSealed decisionsCygnet33.8% correct
- JevUp to 255 options, 64k tokensOptions and input sizeCygnetUp to 26 options, 16,384 tokens
JevBench v1.4.2, measured the same way
One benchmark measured all 93 systems under one method and ranked 89, so these bars are comparable with each other in a way that vendor-published figures are not. The full board and how to read it are on the JevBench results page.
Composite score
The top four finish within 2.3 points of each other, and then the board falls away sharply. The composite hides where systems actually differ, so the two charts below break it apart.
Capability, higher is better
IntelligenceHow often it picks the right answer
CalibrationWhether 0.8 really means about 80%
SpeedMeasured response time
Accuracy by how hard the decision is
Easy72 straightforward cases
Standard96 everyday cases
Judge146 evaluation-style calls
Hard220 genuinely ambiguous cases
Sealed308 private cases, new in v1.4; chance is 29.3%
Easy and standard decisions separate almost nothing. The hard tier, highlighted, is where these systems stop agreeing, and the sealed tier shows how much of that holds on questions nobody could have tuned for.
One thing these charts leave out: the composite score above also weighs a fourth axis, cost. We do not reproduce it here; the full table is at the source below.
Every system on the board
The capability scores are charted above; this is the ranking and the deployment detail behind them. Jev and Cygnet are highlighted.
| Model | Rank | Score | Measured latency | Runs on |
|---|---|---|---|---|
| decider-4b v2 (Mapika, Qwen3.5-4B) | #1 | 64.1 | 0.02s raw, 0.18s adjusted | Self-hosted, RTX PRO 6000 |
| Jev 1.13.0 (TypeSafe) | #2 | 63.3 | 0.65s median / 0.72s p95 | Hosted production API |
| JevK5 v0.2.0 (allebee, Qwen3.5-4B) | #3 | 62.0 | Not re-measured in v1.4 | Self-hosted, RunPod GPU |
| Cygnet (blockbrain, frozen Gemma 4 12B) | #4 | 61.8 | 0.04s raw, 0.22s adjusted | Self-hosted, RTX PRO 6000 |
| Hopper (HopitAI, Qwen3.5-4B LoRA) | #5 | 59.4 | 0.13s raw, 0.41s adjusted | Self-hosted, RTX A6000 |
| Winnow-12B Q8 (EldanRing, Gemma 4 12B) | #6 | 55.6 | 0.23s raw, 0.60s adjusted | Self-hosted, RTX 4090 |
| djev (Maisa, DiffusionGemma) | #8 | 52.2 | 0.24s median / 0.31s p95 | Hosted API, preview |
| SemIf (Qwen3.5-4B) | #11 | 47.7 | 0.20s raw, 0.55s adjusted | Self-hosted, RunPod GPU |
| OpenJev (razorback16) | #27 | 36.9 | 0.24s raw, 0.63s adjusted | Self-hosted, RunPod GPU |
| Laya (ModernBERT-large) | #41 | 30.3 | 0.79s raw, 1.72s adjusted | Self-hosted, CPU |
JevBench v1.4.2, published on 25 September 2026, lists 93 systems and ranks 89 of them. Each is scored on the same 534 public decisions (72 easy, 96 standard, 146 judge, 220 hard) plus 308 sealed decisions whose text stays private, one request at a time. The composite is a harmonic mean of four equally weighted axes (intelligence, calibration, speed and cost), so one weak axis pulls it down hard. Sealed decisions make up 20% of intelligence, and a system that does more than 25 points better on public items than on sealed ones loses intelligence in proportion. Every Jev-style system scores far lower on the sealed set than on the public one, Jev included (36.7% against a 29.3% chance level). Version 1.4.2 kept the v1.4.0 formulas and every earlier measurement; it added 17 systems over two releases, which is why ranks moved while scores did not. Self-hosted and demo latency is adjusted by x2 plus 0.15s to approximate production load, so those numbers are not comparable with the raw latency their authors publish.
Jev and Cygnet feature by feature
| Attribute | Jev 1.13.0 | Cygnet |
|---|---|---|
| What it is | Hosted System One model from TypeSafe, version jev-1.13.0 | Frozen google/gemma-4-12B-it on stock vLLM 0.30.0, plus an MIT shim |
| Licence | Proprietary, hosted | MIT shim; Gemma 4 weights under Apache-2.0 and Google’s prohibited-use policy |
| Request format | POST /v1/systemone | The shim accepts /v1/systemone requests and makes one chat call per decision |
| Question types | Noul, choice, score — many per call, answered in parallel | Noul, choice, score — one request per decision |
| Options per choice | Up to 255 | Up to 26; more is rejected with HTTP 422 |
| Context | 64k tokens per request | 16,384 tokens; longer inputs are rejected |
| Calibration | Post-trained with RLCD; JevBench 76.3 | One temperature, 3.4, fitted on the author’s own items; JevBench 74.9 |
| Hardware | None — it is an API call | A 48 GB GPU in the author’s runs; other cards not measured |
What each one is better at
Where Jev wins
- Sealed-set accuracy 36.7% against 33.8%, and intelligence 53.1 against 49.5.
- Clearly ahead on the sealed set’s reasoning families: trade-offs 38.5% against 19.2%, multi-hop 44.7% against 31.6%.
- 64k tokens per request against 16,384, and up to 255 options per choice against 26.
- Many questions answered in parallel in one call, with nothing to serve.
- A released product with published rate limits, rather than a recipe first published on 24 September.
Where Cygnet wins
- Hard-tier accuracy 75.5% against 74.1%, the best of any top-ten system, and judge tier 95.2% against 94.5%.
- Nothing trained: stock weights, stock vLLM and a shim short enough to read in one sitting.
- Speed 90.7 against 83.3; about 35ms raw per decision on the benchmark’s RTX PRO 6000.
- Ahead of Jev on the sealed set’s hard-judging, adversarial, safety and ambiguous families.
The analysis
Where Jev and Cygnet really differ, and why the bars above land where they do.
Cygnet is the clearest evidence yet that a general model, read the right way, can do most of what a trained decision model does. It places #4 of 89 in JevBench v1.4.2 at 61.8 against Jev’s 63.3, and on the public set it is slightly ahead: 75.5% against 74.1% on the 220 hard decisions and 95.2% against 94.5% on the judge tier. Its speed score is 90.7 against Jev’s 83.3, from a 35ms raw median on the benchmark’s GPU.
The sealed set reverses that. On the 308 decisions nobody could tune for, Jev gets 36.7% right and Cygnet 33.8%, and Cygnet’s gap between public and sealed accuracy, 54.1 points against Jev’s 49.9, costs it on intelligence: 49.5 against 53.1. Calibration is close, 74.9 against 76.3, from one temperature the author fitted on items of their own.
The practical limits are the bigger difference. Cygnet rejects inputs over 16,384 tokens and choices with more than 26 options, and sends one chat request per decision. It has been measured on three GPU models, and the repository appeared on 24 September: it is a recipe you run, not a service.
What Cygnet actually is
Cygnet, typed decisions read from frozen Gemma 4 12B, by blockbrain.
Cygnet is a recipe from blockbrain. It runs google/gemma-4-12B-it at a pinned revision on unmodified vLLM 0.30.0, with a small shim in front, written with the Python standard library, that accepts JevBench’s /v1/systemone requests. Nothing about the model is trained or merged.
For each decision the shim sends one chat request with the state, the instructions and the options relabelled A, B, C and so on, and asks for one letter. vLLM restricts the answer position to those letters and returns their log-probabilities; the shim sums every token that decodes to each letter, renormalises over the options and applies a calibration temperature of 3.4. Output is one token per decision. The author credits the one-token readout to NInfer.
The temperature was fitted on 241 items the author generated; the README states that JevBench’s public items were used only to measure, never to fit. On the public set the author measured 203 of 231 correct on both an RTX A6000 and an L40S, with a 50–66ms median on the standard tier.
The shim is MIT. The weights are Google’s Gemma 4 12B under Apache-2.0, subject to Google’s Gemma Prohibited Use Policy.
Where each one wins on the sealed set
Cygnet wins the judging families: hard judging, 53.7% against Jev’s 34.1%; trap and adversarial items, 50.0% against 41.7%; safety judgments, 43.8% against 37.5%; and ambiguous-abstain items, 37.8% against 29.7%. The two tie on paraphrase robustness at 64.3%.
Jev wins every reasoning family: probability questions, 50.0% against 35.7%; multi-hop lookups, 44.7% against 31.6%; trade-offs, 38.5% against 19.2%; dates and numbers, 28.6% against 17.9%; and long policies, 27.5% against 22.5%. Every system of this kind scores low on the sealed set overall, not far above the 29.3% chance level, so treat these as a guide to what to test rather than a verdict.
So which should you use?
Pick Jev if
- Your hard cases involve numbers, dates, trade-offs or multi-step lookups.
- Inputs are long, or a choice has more than 26 options.
- You want a hosted, versioned model rather than a GPU and a shim to operate.
Pick Cygnet if
- Your decisions are judgments about text, safety or policy compliance, where Cygnet is strongest.
- You already run Gemma 4 on vLLM, or want a decision layer with no training step to reproduce.
- Inputs fit in 16k tokens and choices have at most 26 options.
Try Jev on the cases you are actually arguing about
Cygnet and Jev split the sealed families cleanly, judging for Cygnet and reasoning for Jev, so the comparison that settles it is on your own cases. Run them on Jev here with no setup and keep the answers; they show which side of that split your decisions fall on.
Jev vs Cygnet FAQ
Is Cygnet better than Jev?
Not overall: JevBench v1.4.2 ranks it #4 at 61.8 against Jev’s 63.3. It is slightly ahead on the public hard and judge tiers, 75.5% against 74.1% and 95.2% against 94.5%, and behind on the sealed set, 33.8% against 36.7%.
Is Cygnet fine-tuned?
No. It runs Google’s gemma-4-12B-it unchanged on stock vLLM. The only fitted value is one calibration temperature, 3.4, which the author fitted on 241 self-generated items rather than on JevBench’s public set.
What hardware does Cygnet need?
The author measured it on an RTX A6000 and an L40S, both 48 GB, and JevBench measured it on an RTX PRO 6000. Other cards have not been measured.
Is Cygnet related to TypeSafe or to Winnow?
Neither. It is an independent recipe from blockbrain. Winnow also uses Gemma 4 12B but fine-tunes it; Cygnet uses the stock model.
Other Jev comparisons
Every number on these pages is quoted from a published source and was read on September 26, 2026.
- Jev vs decider-4bThe new first place on JevBench, and what that first place is made of.
- Jev vs JevK5One of three open systems in JevBench’s top four, and where the gap to Jev still is.
- Jev vs WinnowOne local model for decisions, chat and images, against a hosted decision model.
- Jev vs djevA faster decision API, against a better calibrated one.
- Jev vs SemIfA fast open rebuild on your own GPU, and where it still falls short.
- Jev vs OpenJevOne name, ten projects. Here is every one, and how the drop-in compares.
- Jev vs LayaA model you fine-tune on your data, against one that works out of the box.
Sources
Cygnet and every other product named on this page belongs to its respective owner and is not affiliated with Jev AI. Figures are quoted from the sources above as they read on September 26, 2026 and can change.