On this site
Jev 1.13.0
TypeSafe · System One model
Ahead on hard cases, the sealed set and calibration.
63.3JevBench v1.4.0 rank #1
Jev comparison · JevBench v1.4.0 · updated September 24, 2026
Jev vs Winnow
One local model for decisions, chat and images, against a hosted decision model.
vs
Alternative
Winnow
EldanRing · Gemma 4 12B decision fine-tune
Decisions, chat and vision from one model on a 16 GB GPU.
55.6JevBench v1.4.0 rank #4
- JevHosted API, generally availableWhere it runsWinnowSelf-hosted GGUF; Q8 tested on a 16 GB RTX 5070 Ti
- JevTyped decisions onlyWhat it doesWinnowTyped decisions, chat and image input from one server
- Jev94.5% correctEvaluation-style casesWinnow91.1% correct
- Jev74.1% correctHardest test casesWinnow70.9% correct
- Jev36.7% correctSealed decisionsWinnow33.1% correct
- JevCalibrated — 76.3 on JevBenchProbabilitiesWinnowNormalised logits, no fitted calibration — 64.8
JevBench v1.4.0, measured the same way
One benchmark ran all 52 systems under one method, so these bars are comparable with each other in a way that vendor-published figures are not. Version 1.3.0 began measuring intelligence above chance, which moved several systems tens of places without any of them changing.
Composite score
The top three finish within 1.4 points of each other, and then the board falls away sharply. The composite hides where systems actually differ, so the two charts below break it apart.
Capability, higher is better
IntelligenceHow often it picks the right answer
CalibrationWhether 0.8 really means about 80%
SpeedMeasured response time
Accuracy by how hard the decision is
Easy72 straightforward cases
Standard96 everyday cases
Judge146 evaluation-style calls
Hard220 genuinely ambiguous cases
Sealed308 private cases, new in v1.4; chance is 29.3%
Easy and standard decisions separate almost nothing. The hard tier, highlighted, is where these systems stop agreeing, and the sealed tier shows how much of that holds on questions nobody could have tuned for.
One thing these charts leave out: the composite score above also weighs a fourth axis, cost. We do not reproduce it here; the full table is at the source below.
Every system on the board
The capability scores are charted above; this is the ranking and the deployment detail behind them. Jev and Winnow are highlighted.
| Model | Rank | Score | Measured latency | Runs on |
|---|---|---|---|---|
| Jev 1.13.0 (TypeSafe) | #1 | 63.3 | 0.65s median / 0.72s p95 | Hosted production API |
| JevK5 v0.2.0 (allebee, Qwen3.5-4B) | #2 | 62.0 | Not re-measured in v1.4 | Self-hosted, RunPod GPU |
| Hopper (HopitAI, Qwen3.5-4B LoRA) | #3 | 59.4 | 0.13s raw, 0.41s adjusted | Self-hosted, RTX A6000 |
| Winnow-12B Q8 (EldanRing, Gemma 4 12B) | #4 | 55.6 | 0.23s raw, 0.60s adjusted | Self-hosted, RTX 4090 |
| djev (Maisa, DiffusionGemma) | #6 | 52.2 | 0.24s median / 0.31s p95 | Hosted API, preview |
| SemIf (Qwen3.5-4B) | #8 | 47.7 | 0.20s raw, 0.55s adjusted | Self-hosted, RunPod GPU |
| OpenJev (razorback16) | #21 | 36.9 | 0.24s raw, 0.63s adjusted | Self-hosted, RunPod GPU |
| Laya (ModernBERT-large) | #32 | 30.3 | 0.79s raw, 1.72s adjusted | Self-hosted, CPU |
JevBench v1.4.0 scored 76 systems, 71 of them ranked, on 23 September 2026: the same 534 public decisions as before (72 easy, 96 standard, 146 judge, 220 hard) plus 308 sealed decisions whose text stays private, one request at a time. Version 1.4.0 changed the score in three ways. The composite is now a harmonic mean of four equally weighted axes (intelligence, calibration, speed and cost), so one weak axis pulls it down much harder than before. Sealed decisions make up 20% of intelligence, and a system that does more than 25 points better on public items than on sealed ones loses intelligence in proportion. Calibration now blends in the sealed results. Every Jev-style system scores far lower on the sealed set than on the public one, Jev included (36.7% against a 29.3% chance level), so every composite fell; the public tier accuracies are unchanged from v1.3.0. Self-hosted and demo latency is adjusted by x2 plus 0.15s to approximate production load, so those numbers are not comparable with the raw latency their authors publish.
Jev and Winnow feature by feature
| Attribute | Jev 1.13.0 | Winnow |
|---|---|---|
| What it is | Hosted System One model from TypeSafe, version jev-1.13.0 | Gemma 4 12B with a merged LoRA fine-tune, released as GGUF |
| Licence | Proprietary, hosted | Apache-2.0, with the Gemma 4 terms that apply to derivatives |
| Request format | POST /v1/systemone | /v1/systemone for decisions and /v1/chat/completions for chat, from one server |
| Question types | Noul, choice, score — many per call, answered in parallel | Noul, choice, score — one branch per question over shared state |
| Inputs | Text only: string, JSON object or array | Text, plus images with the optional vision projector |
| Context | 64k tokens per request | 65,536 positions tested with vision on a 16 GB GPU |
| Calibration | Post-trained with RLCD; JevBench 76.3 | Entropy-based confidence, no fitted calibration map; JevBench 64.8 |
| Training data | Undisclosed | Private; categories described in the model card |
| Hardware | None — it is an API call | A 16 GB GPU for the Q8 build; the BF16 build needs more memory or offloading |
What each one is better at
Where Jev wins
- Hard-tier accuracy 74.1% against 70.9%, judge tier 94.5% against 91.1%, and sealed set 36.7% against 33.1%.
- Calibration 76.3 against 64.8; Winnow’s confidence is documented as entropy-based rather than a probability of being right.
- Clearly ahead on the sealed set’s probability, trade-off, date-and-number and multi-hop families.
- Nothing to download, serve or tune: no 12.7 GB model file, no GPU scheduling between chat and decision contexts.
Where Winnow wins
- One model for typed decisions, chat and image input, served from the same process.
- Ahead of Jev on the sealed set’s hard-judging, safety-judging, adversarial and ambiguous families.
- Runs locally on one 16 GB consumer GPU with a 64K context, and ties Jev on a 231-item public subset by its author’s measurement.
- Apache-2.0 GGUF weights you can run offline or inside your own network.
The analysis
Where Jev and Winnow really differ, and why the bars above land where they do.
Winnow is the strongest system on the board that also chats and reads images. JevBench v1.4.0 ranks it #4 of 71 at 55.6 against Jev’s 63.3. It matches Jev on every easy decision, and its author reports that the 8-bit build ties Jev on a 231-item public JevBench subset, 198 correct each, while stating that this is a result on that subset and not a claim of parity.
Jev keeps the lead where it counts for most decision pipelines: 74.1% against 70.9% on the 220 hard decisions, 94.5% against 91.1% on the judge tier, and 36.7% against 33.1% on the 308 sealed ones. Calibration is the widest gap, 76.3 against 64.8. Winnow’s own model card says its confidence is entropy-based and not a guaranteed probability of correctness, with no separately fitted calibration.
Winnow’s case is breadth on your own hardware. The same loaded model serves /v1/systemone for decisions and /v1/chat/completions for chat, takes images through an optional vision projector, and was tested with a 64K context on a single consumer GPU. If you want one local model to decide, explain and look at a screenshot, nothing else on this site does all three.
What Winnow actually is
Winnow-12B, the Gemma 4 decision fine-tune from EldanRing, by EldanRing.
Winnow-12B is a LoRA fine-tune of google/gemma-4-12B-it, merged into the weights and released under Apache-2.0 as two GGUF files: an 8-bit Q8_0 build of about 12.7 GB, recommended for 16 GB GPUs, and a 16-bit BF16 build of about 23.8 GB. A separate 175 MB projector adds image input. It runs in the author’s llama.cpp-based inference server; no conversion step or separate adapter is needed.
For decisions, the server fills in the shared state once, forks one branch per question, and reads the logits of the answer tokens without generating any text, the same idea as Jev’s typed questions. It supports noul, choice and score. It states that it is not affiliated with or endorsed by TypeSafe.
The training data is private. The model card describes curated typed-decision examples covering routing, policy and rule application, evidence selection, workflow decisions, ordinal judgments, entailment, paraphrase and answerability, followed by a refinement pass. It makes no claim that public benchmarks were excluded from the base model’s pretraining data.
On the author’s RTX 5070 Ti test, a four-question request at near-full 64K context took 25 seconds cold and a median of 143ms once the context was cached, with peak GPU memory of 15.01 GiB including the desktop. JevBench measured it on an RTX 4090 at 0.23s median per decision before its self-hosting adjustment.
Where each one wins on the sealed set
On the 308 sealed decisions, grouped by family, Winnow wins the judging families. It gets 46.3% of hard judging items against Jev’s 34.1%, 50.0% of safety judgments against 37.5%, 50.0% of trap and adversarial items against 41.7%, and 35.1% of ambiguous-abstain items against 29.7%. The two tie on paraphrase robustness at 64.3%.
Jev wins the reasoning families: probability questions, 50.0% against 32.1%; trade-offs, 38.5% against 23.1%; dates and numbers, 28.6% against 16.1%; multi-hop lookups, 44.7% against 36.8%; and long policies, 27.5% against 22.5%. Every system of this kind scores low on the sealed set overall, not far above the 29.3% chance level, so treat these family results as a guide to what to test rather than a verdict.
A second benchmark from the author
Winnow’s model card also reports a second suite, Kev-v9 clean, with 1,046 items, run by its author with Jev called through OpenRouter on the same inputs and scoring rules. There Jev scores 87.00% against 81.55% for Winnow Q8. On the 231-item public JevBench subset the author reports a tie at 85.71%. These are the author’s measurements, not JevBench’s official composite.
So which should you use?
Pick Jev if
- You branch on the probability and need it to mean something without fitting a calibration map yourself.
- Your hard cases involve numbers, dates, trade-offs or multi-step lookups.
- You want a hosted, versioned model with published rate limits rather than a GPU to operate.
Pick Winnow if
- You want a single local model that decides, chats and reads screenshots or photos.
- Your data cannot leave your network and you have a 16 GB GPU to spare.
- Your decisions are mostly judgments about text or safety, where Winnow is strongest.
Try Jev on the cases you are actually arguing about
Winnow and Jev split the sealed families almost down the middle, so the useful comparison is on your own cases. Run them on Jev here — it takes a minute with no setup — and keep the answers; if Winnow agrees on the decisions you care about, a local model that also chats and reads images may be the better fit.
Jev vs Winnow FAQ
Is Winnow as accurate as Jev?
Not overall. JevBench v1.4.0 ranks Winnow #4 at 55.6 against Jev’s 63.3. Jev is ahead on hard decisions, 74.1% against 70.9%, and on the sealed set, 36.7% against 33.1%. Winnow’s author reports a tie on a 231-item public subset and a Jev lead on a separate 1,046-item suite.
Can Winnow read images?
Yes. With the optional 175 MB vision projector it accepts images through the same server. Jev accepts text only.
What GPU do I need for Winnow?
The author tested the 8-bit Q8_0 build, about 12.7 GB, with full offload on a 16 GB RTX 5070 Ti at a 64K context, peaking at 15.01 GiB. The 16-bit build is about 23.8 GB and needs more GPU memory or offloading.
Are Winnow’s probabilities calibrated?
Not in the way Jev’s are. Its model card says the server normalises logits over the options you supply and that its entropy-based confidence is not a guaranteed probability of correctness, with no separately fitted calibration map. JevBench scores its calibration at 64.8 against Jev’s 76.3.
Is Winnow affiliated with TypeSafe?
No. It is an independent fine-tune of Gemma 4 12B by EldanRing, and its model card states it is not affiliated with or endorsed by TypeSafe or Google.
Other Jev comparisons
Every number on these pages is quoted from a published source and was read on September 24, 2026.
- Jev vs JevK5The closest open system to Jev on the board, and where the gap still is.
- Jev vs djevA faster decision API, against a better calibrated one.
- Jev vs SemIfA fast open rebuild on your own GPU, and where it still falls short.
- Jev vs OpenJevOne name, ten projects. Here is every one, and how the drop-in compares.
- Jev vs LayaA model you fine-tune on your data, against one that works out of the box.
Sources
Winnow and every other product named on this page belongs to its respective owner and is not affiliated with Jev AI. Figures are quoted from the sources above as they read on September 24, 2026 and can change.