Measured here · 396 calls · ran 2026-09-20
Jev latency and cost, measured on the same task
Every cost figure circulating for Jev was produced on a different task, in a different warm state, against a different baseline. These numbers were produced on one task set, and the method is written down.
- Calls
- 396
- 0 failed
- Warm samples
- 31 / cell
- 2 passes
- Cold idle wait
- 90s
- before the retest
- Concurrency
- 8
- single process
What this run found
- Jev was not the fastest: warm p50 312.7 ms against 263.4 ms and 258.8 ms for the baselines.
- Its cost advantage over the cheap baseline was under 2× ($0.0200 vs $0.0351 per 1,000 decisions), and about 7× against the frontier baseline.
- It used ~2.3× more input tokens (475.6 vs 209.6 per decision) — a structured state plus a criteria map is more verbose than a sentence.
- From this location a minimal request already costs 281–944 ms, and Jev's floor is essentially equal to its task latency. A 200× speed advantage cannot be demonstrated here.
Latency
Warm p50 / p95 over 31 samples per cell. Bar length encodes magnitude only — it is not a quality judgement, and cold is shown because a cold call costs more than a frame at 60 fps.
| Model | Warm p50 | Warm p95 | Cold | After idle | Input tok/decision |
|---|---|---|---|---|---|
| Jev (TypeSafe) | 312.7 | 823.3 | 443.3 | 605.1 | 475.6 |
| DeepSeek V4.1 Flash | 263.4 | 558.8 | 348.7 | 321.3 | 209.6 |
| DeepSeek V4 Pro | 258.8 | 327.8 | 255.3 | 257.8 | 208.6 |
Lower is better. Jev's warm median is the highest of the three, and its p95 is more than twice the frontier baseline's — on this workload and from this location.
Round-trip floor
Minimal-payload requests, 20 each. This is not a model floor — the chat endpoints' floor is higher than their own task latency, because the two go through different queueing paths. Its only use is to show how much of a single call is spent on the wire.
| Model | Serial p50 | Serial p95 | Concurrent p50 | Task − floor |
|---|---|---|---|---|
| Jev (TypeSafe) | 281.8 | 336.8 | 343.8 | +30.9 |
| DeepSeek V4.1 Flash | 704.8 | 936.7 | 792.1 | -441.4 |
| DeepSeek V4 Pro | 932.3 | 1219.1 | 944 | -673.5 |
Jev's task latency is almost exactly its floor — on this workload the round trip dominates. Both baselines undercut their own floor, so no latency conclusion follows from these rows alone.
Disagreement
Paired on the same item, warm only, both calls OK.
jev vs deepseek-flash
3.1%
2 of 64 pairs
first wrong only 0 · second wrong only 2
jev vs deepseek-v4-pro
7.8%
5 of 64 pairs
first wrong only 2 · second wrong only 3
Confidence and the fallback tail
Only a decision model returns a calibrated confidence, so only it can be thresholded. The share below 0.90 is the fraction a threshold-based pipeline would hand to a larger model — that tail is what the routing economics actually depend on.
Confidence p50
0.99
Confidence mean
0.91
Below 0.90
19.4%
Warm decisions
124
| Task | Confidence p50 | Below 0.90 | Accuracy |
|---|---|---|---|
| topic-classify | 1.00 | 0.0% | 100.0% |
| tool-route | 1.00 | 3.2% | 100.0% |
| policy-violation | 0.99 | 25.8% | 83.9% |
| move-select | 0.98 | 48.4% | 100.0% |
Why this page exists
Publicly reported per-item costs for Jev differ by more than an order of magnitude. None of them states document length, label-set size, token counts, or whether the calls were warm. A number without its method is not comparable to anything.
So we ran the same four decision tasks against Jev and two chat models, published every raw response, and wrote down what the numbers can and cannot support. The short version: on these tasks and from this location, Jev was not the fastest, and the cost gap against a cheap chat model was under 2x — nothing like the 40–400x figures circulating.
endpoint POST https://api.typesafe.ai/v1/systemone
model jev-latest (reported by the API as jev-1.13.0 on 2026-09-20)
answers { choice | score, confidence, probabilities }
usage { input_tokens, output_tokens }Method
Four decision tasks over a fixed 16-item input set each: a six-way topic classification, a ten-way tool route, a policy check with a severity score, and a move selection where the best answer is computed from an explicit scoring rule. Every task, item and expected answer is in the repository.
Warm and cold calls are reported separately. Cold is one sample per task per model, plus a second cold pass after 90 seconds of idle. Warm latency is reported as p50 and p95 over 31 samples per cell, using nearest-rank. Cost per 1,000 decisions is each vendor's own token count times its own published price. Each configuration ran in one process at concurrency 8 from Hong Kong.