Jev latency and cost, measured on the same task

Why this page exists

Publicly reported per-item costs for Jev differ by more than an order of magnitude. None of them states document length, label-set size, token counts, or whether the calls were warm. A number without its method is not comparable to anything.

So we ran the same four decision tasks against Jev and two chat models, published every raw response, and wrote down what the numbers can and cannot support. The short version: on these tasks and from this location, Jev was not the fastest, and the cost gap against a cheap chat model was under 2x — nothing like the 40–400x figures circulating.

endpoint  POST https://api.typesafe.ai/v1/systemone
model     jev-latest            (reported by the API as jev-1.13.0 on 2026-09-20)
answers   { choice | score, confidence, probabilities }
usage     { input_tokens, output_tokens }

Method

Four decision tasks over a fixed 16-item input set each: a six-way topic classification, a ten-way tool route, a policy check with a severity score, and a move selection where the best answer is computed from an explicit scoring rule. Every task, item and expected answer is in the repository.

Warm and cold calls are reported separately. Cold is one sample per task per model, plus a second cold pass after 90 seconds of idle. Warm latency is reported as p50 and p95 over 31 samples per cell, using nearest-rank. Cost per 1,000 decisions is each vendor's own token count times its own published price. Each configuration ran in one process at concurrency 8 from Hong Kong.

What was run

Ran 2026-09-20T08:15:47.076Z. API-reported model versions: jev-1.13.0, deepseek-flash, deepseek-v4-pro.

Per-model summary

Warm latency only in p50/p95. Cold columns are a single sample per task (n = 4), so read them as existence proofs, not distributions.

Accuracy by task

Correct answers out of warm samples. Ground truth for three of the four tasks is our own labelling — that is a judgement, not a fact. Only move-select is mechanically checkable.

• topic-classify — ground truth: 本站人工标注（是判断，不是事实）
• tool-route — ground truth: 本站人工标注（是判断，不是事实）
• policy-violation — ground truth: 本站人工标注（是判断，不是事实）；severity 分数是主观标注
• move-select — ground truth: **机械可算**：按 gain − cost − risk 取最大值（唯一不含主观标注的任务）

Latency by task

Warm p50 / p95 in milliseconds. p95 uses nearest-rank, so with 31 warm samples per cell it is indicative rather than precise.

Jev confidence and the fallback tail

Only Jev returns a calibrated confidence, so only Jev can be thresholded. The share below 0.9 is the fraction of decisions a threshold-based pipeline would hand to a larger model — that tail is what the routing economics actually depend on.

Where the models disagree

Paired on the same item, warm answers, both calls succeeding. The two error columns are the asymmetry worth watching: how often one model was wrong and the other right.

Round-trip floor: how much of this is network

Minimal-payload requests (a two-option decision with a one-field state for Jev; a one-token reply with thinking disabled for the chat models), n = 20 each, measured 2026-09-20. This is not a model floor — see the caveat below — it exists to show the scale of the round trip before any model work happens.

Minimal-payload round trip. Not a model benchmark: it exists to show how much of a single-request latency at this location is network and queueing rather than compute.

What these numbers are not

• Token counts are not comparable across vendors. Each figure comes from that vendor's own usage field. Cost is computed as each vendor's own count times each vendor's own published price, which is the operationally meaningful comparison but not a like-for-like token count.
• Cold is n = 4 per model. One sample per task. Do not read a p-value into it.
• The biggest confound is where these calls were made from. Every number here was produced from a machine in Hong Kong. The floor table shows a minimal-payload round trip already costs hundreds of milliseconds to each endpoint, and for Jev that floor is essentially equal to its task latency — meaning at this distance a single request is dominated by the round trip, not by model compute. A benchmark cannot prove a 200× speed advantage under that condition. Run the same scripts from your own region before drawing a latency conclusion.
• These four tasks are too easy to separate the models. Accuracy is at or near ceiling for all three, so accuracy here says almost nothing. The 200×/400× claims circulating on X are about different workloads than these; this page measures latency, cost and agreement on four bounded decision shapes, and nothing else.
• Baselines ran with thinking disabled. DeepSeek models default to a reasoning mode that emits chain-of-thought before the answer, which would both inflate their latency and cost and make the comparison unfair against a single-pass decision model. Their numbers here are the non-thinking configuration.
• Pricing moves. Prices were cited on 2026-09-20; each row carries its source below.

Raw responses (every call, including failures) are published with this repository as data/bench-raw.json; the task set is scripts/bench/tasks.mjs. Recompute rather than trust.