On this site
Jev 1.13.0
TypeSafe · System One model
Still first overall, and ahead on hard cases and the sealed set.
63.3JevBench v1.4.0 rank #1
Jev comparison · JevBench v1.4.0 · updated September 24, 2026
Jev vs JevK5
The closest open system to Jev on the board, and where the gap still is.
vs
Alternative
JevK5
allebee · Open Qwen3.5-4B with a distilled LoRA
Second of 76, and the weights are yours to run.
62.0JevBench v1.4.0 rank #2
- JevHosted API, generally availableWhere it runsJevK5Self-hosted on your GPU, about 9GB in bf16
- JevProprietary, hostedLicenceJevK5Apache-2.0 weights and code
- Jev94.5% correctEvaluation-style casesJevK594.5% correct
- Jev74.1% correctHardest test casesJevK570.0% correct
- Jev36.7% correctSealed decisionsJevK533.1% correct
- JevUp to 255 options, 64k tokensOptions and input sizeJevK5Up to 16 options, 16,384 tokens
JevBench v1.4.0, measured the same way
One benchmark ran all 52 systems under one method, so these bars are comparable with each other in a way that vendor-published figures are not. Version 1.3.0 began measuring intelligence above chance, which moved several systems tens of places without any of them changing.
Composite score
The top three finish within 1.4 points of each other, and then the board falls away sharply. The composite hides where systems actually differ, so the two charts below break it apart.
Capability, higher is better
IntelligenceHow often it picks the right answer
CalibrationWhether 0.8 really means about 80%
SpeedMeasured response time
Accuracy by how hard the decision is
Easy72 straightforward cases
Standard96 everyday cases
Judge146 evaluation-style calls
Hard220 genuinely ambiguous cases
Sealed308 private cases, new in v1.4; chance is 29.3%
Easy and standard decisions separate almost nothing. The hard tier, highlighted, is where these systems stop agreeing, and the sealed tier shows how much of that holds on questions nobody could have tuned for.
One thing these charts leave out: the composite score above also weighs a fourth axis, cost. We do not reproduce it here; the full table is at the source below.
Every system on the board
The capability scores are charted above; this is the ranking and the deployment detail behind them. Jev and JevK5 are highlighted.
| Model | Rank | Score | Measured latency | Runs on |
|---|---|---|---|---|
| Jev 1.13.0 (TypeSafe) | #1 | 63.3 | 0.65s median / 0.72s p95 | Hosted production API |
| JevK5 v0.2.0 (allebee, Qwen3.5-4B) | #2 | 62.0 | Not re-measured in v1.4 | Self-hosted, RunPod GPU |
| Hopper (HopitAI, Qwen3.5-4B LoRA) | #3 | 59.4 | 0.13s raw, 0.41s adjusted | Self-hosted, RTX A6000 |
| djev (Maisa, DiffusionGemma) | #6 | 52.2 | 0.24s median / 0.31s p95 | Hosted API, preview |
| SemIf (Qwen3.5-4B) | #8 | 47.7 | 0.20s raw, 0.55s adjusted | Self-hosted, RunPod GPU |
| OpenJev (razorback16) | #21 | 36.9 | 0.24s raw, 0.63s adjusted | Self-hosted, RunPod GPU |
| Laya (ModernBERT-large) | #32 | 30.3 | 0.79s raw, 1.72s adjusted | Self-hosted, CPU |
JevBench v1.4.0 scored 76 systems, 71 of them ranked, on 23 September 2026: the same 534 public decisions as before (72 easy, 96 standard, 146 judge, 220 hard) plus 308 sealed decisions whose text stays private, one request at a time. Version 1.4.0 changed the score in three ways. The composite is now a harmonic mean of four equally weighted axes (intelligence, calibration, speed and cost), so one weak axis pulls it down much harder than before. Sealed decisions make up 20% of intelligence, and a system that does more than 25 points better on public items than on sealed ones loses intelligence in proportion. Calibration now blends in the sealed results. Every Jev-style system scores far lower on the sealed set than on the public one, Jev included (36.7% against a 29.3% chance level), so every composite fell; the public tier accuracies are unchanged from v1.3.0. Self-hosted and demo latency is adjusted by x2 plus 0.15s to approximate production load, so those numbers are not comparable with the raw latency their authors publish.
Jev and JevK5 feature by feature
| Attribute | Jev 1.13.0 | JevK5 |
|---|---|---|
| What it is | Hosted System One model from TypeSafe, version jev-1.13.0 | Open-weight Qwen3.5-4B with a merged, distilled LoRA |
| Licence | Proprietary, hosted | Apache-2.0 for weights and code |
| Request format | POST /v1/systemone | Its own server accepts the TypeSafe-style /v1/systemone request shape |
| Question types | Noul, choice, score — many per call, answered in parallel | Noul, choice, score — each question evaluated separately |
| Options per choice | Up to 255 | Up to 16 |
| Context | 64k tokens per request | Refuses inputs over 16,384 tokens |
| Languages | Best in English; other languages work with lower accuracy | English only |
| Calibration | Post-trained with RLCD; JevBench 76.3 | One temperature fitted on held-out questions; JevBench 74.5 |
| Hardware | None — it is an API call | Your own GPU, about 9GB in bf16; the author’s figures are on an H100 and an L40S |
What each one is better at
Where Jev wins
- Hard-tier accuracy 74.1% against 70.0%, and sealed-set accuracy 36.7% against 33.1%.
- Paraphrase robustness on the sealed set, 64.3% against 14.3%: reword the question and Jev’s answer holds far more often.
- 64k tokens per request against a 16,384-token limit, and up to 255 options per choice against 16.
- Nothing to operate, and questions in one call answered in parallel rather than serialized on one GPU.
- Confidence usable across tiers as delivered; JevK5’s README reports standard-tier confidence running too low after its temperature fit.
Where JevK5 wins
- Second of 76 in JevBench v1.4.0 and the highest-ranked open system: 62.0 against Jev’s 63.3.
- Ties Jev on the judge tier at 94.5%, and matches it on every easy decision.
- Speed 91.1 against Jev’s 83.3; the author measures about 13ms per short decision on an H100.
- Apache-2.0 weights you can run offline or inside your own network, with the training recipe published.
- Ahead of Jev on the sealed set’s ambiguous and adversarial families.
The analysis
Where Jev and JevK5 really differ, and why the bars above land where they do.
JevK5 is the first open system to come within touching distance of Jev on JevBench. In v1.4.0 it places #2 of 71 at 62.0 against Jev’s 63.3, the highest of any open system, and it ties Jev on the 146-decision judge tier at 94.5%. On speed it scores 91.1 against Jev’s 83.3; its author reports about 13ms per short decision on an H100, measured in-process rather than over a network.
Jev keeps the edge where the cases get hard: 74.1% against 70.0% on the 220 hard decisions, 36.7% against 33.1% on the 308 sealed ones, and 53.1 against 48.9 on intelligence. Calibration is close, 76.3 against 74.5. JevK5 gets there with a single fitted temperature, and its README notes that confidence on standard-tier questions comes out too low as a result.
The other differences are limits rather than scores. JevK5 is English-only, handles up to 16 options per choice, refuses inputs over 16,384 tokens and serializes requests on one GPU. It is also new: the repository appeared on 22 September, version 0.2.0 followed, and its author says its quality on Jev’s published real-world workflows has not been measured yet.
What JevK5 actually is
JevK5, the open-weight Jev alternative from allebee, by allebee.
JevK5 is Qwen3.5-4B with a distilled LoRA merged into the weights, published on Hugging Face as alibiserikbay/JevK5 under Apache-2.0, with the code on GitHub. It reads answers the way SemIf does: a softmax over the next-token logits of the answer letters with one temperature, so every option gets a probability in a single forward pass and no text is generated. It states that it is not affiliated with TypeSafe, and it does not reproduce Jev’s unpublished architecture.
The training is what separates it from SemIf. Qwen3.6-27B, with thinking, wrote documents carrying hard typed questions across 17 business domains and 11 decision families, answered each question twice, and kept it only when both answers matched. JevK5 was trained on 3,272 of those questions plus 3,272 human-labelled items from public datasets such as MMLU-Pro, BoolQ and banking77. The author states that no JevBench item and no Jev output was used for training.
The author also documents a correction made on 23 September: a small hand-written set used to choose the calibration setting echoed the wording of a few public benchmark items. It was rewritten, and the author reports that no accuracy figure changed.
It installs from GitHub as a Python package and ships a server that accepts the TypeSafe-style /v1/systemone request shape. In bf16 it needs about 9GB of GPU memory. On one H100 the author measures a 13.5ms median on easy and standard items and a 30ms median on hard items with 1–4k-token documents.
Where each one wins on the sealed set
The sealed set is 308 decisions nobody outside the benchmark has seen, grouped into families, and the two models win different ones. JevK5 is ahead on ambiguous questions where abstaining is the right call, 43.2% against 29.7%, and on trap and adversarial items, 58.3% against 41.7%.
Jev is far ahead on paraphrase robustness, 64.3% against 14.3%: when the same question is reworded, its answer holds much more often. It also leads on probability questions, 50.0% against 39.3%, trade-offs, 38.5% against 23.1%, and multi-hop lookups, 44.7% against 39.5%.
Both models score low on the sealed set overall, not far above the 29.3% chance level, and so does every other system of this kind; JevBench’s author calls the set unusually difficult. Treat these family numbers as a guide to what to test on your own data, not as a verdict.
So which should you use?
Pick Jev if
- Inputs are long documents, traces or tickets over 16k tokens, or choices with more than 16 options.
- The same question gets worded differently by different people or systems, where Jev’s answers hold more consistently.
- You want a hosted, versioned model with published rate limits instead of a new project to operate.
Pick JevK5 if
- You need the weights: offline use, data residency, or a model you can inspect and fine-tune.
- Your inputs are English, fit in 16k tokens and have at most 16 options.
- You already run GPUs and want millisecond-level decisions on your own hardware.
Try Jev on the cases you are actually arguing about
JevK5 is close enough that the honest comparison is on your own cases. Run them on Jev here — it takes a minute with no setup — and keep the answers: if JevK5 agrees on the cases you care about, you have a strong reason to self-host, and if it does not, you know exactly where the gap is.
Jev vs JevK5 FAQ
Is JevK5 better than Jev?
Not overall. JevBench v1.4.0 ranks it second, 62.0 against Jev’s 63.3. It ties Jev on the judge tier and is faster, but trails on hard decisions, 70.0% against 74.1%, and on the sealed set, 33.1% against 36.7%.
Is JevK5 made by TypeSafe?
No. It is an independent Apache-2.0 project by allebee and states that it is not affiliated with TypeSafe AI. It uses open Qwen3.5-4B weights and SemIf’s option-logit readout rather than Jev’s architecture.
Can I send Jev requests to JevK5?
Its server accepts the TypeSafe-style /v1/systemone request shape, so requests written for Jev need little or no change. The project does not claim full SDK compatibility, and it caps choices at 16 options and inputs at 16,384 tokens, so test your own requests before switching.
What hardware does JevK5 need?
About 9GB of GPU memory in bf16. The author’s figures are on an H100, 13.5ms median on easy and standard items, and an L40S, about 21ms for a demo decision. Without the optional flash-linear-attention package, long inputs run slower.
How is JevK5 different from SemIf?
It uses SemIf’s readout on the same Qwen3.5-4B base, then adds a LoRA distilled from Qwen3.6-27B on double-checked hard questions and a fitted temperature. JevBench v1.4.0 ranks JevK5 #2 and SemIf #8, and on hard decisions JevK5 gets 70.0% against SemIf’s 59.5%.
Other Jev comparisons
Every number on these pages is quoted from a published source and was read on September 24, 2026.
- Jev vs djevA faster decision API, against a better calibrated one.
- Jev vs SemIfA fast open rebuild on your own GPU, and where it still falls short.
- Jev vs OpenJevOne name, ten projects. Here is every one, and how the drop-in compares.
- Jev vs LayaA model you fine-tune on your data, against one that works out of the box.
Sources
JevK5 and every other product named on this page belongs to its respective owner and is not affiliated with Jev AI. Figures are quoted from the sources above as they read on September 24, 2026 and can change.