Jev · lab-v0.1
Jev Benchmark Lab
Compare systems by metric with mandatory source-kind labels. Vendor claims, independent publications and TenthMatrix methodology demos are never mixed silently.
方法论
Rows tagged vendor_reported quote TypeSafe (or partners) with links. Rows tagged independent quote third-party publications. Rows tagged workstation_methodology illustrate harness wiring and must not be cited as TenthMatrix-measured production results until a suite is promoted to workstation_measured in a later lab version.
A — Intent / triage (cited public results)
Public triage/intent-style measurements from OpenRouter and AyAutomate, plus TypeSafe’s qualitative positioning. Not a single merged leaderboard.
- cite-2026-09-19
- 19–20 Sep 2026
- 样本数:60–791 depending on source
| 系统 | Source | p50 latency (ms) | Accuracy (%) | Cost per 1k tasks (USD) |
|---|---|---|---|---|
| Jev 1.13 (OpenRouter triage) | 独立测评OpenRouter tutorial tables | 194 ms p50 | 59/60 intent (98.3%) | $0.0248 / 1k (author table) |
| GPT Luna (OpenRouter triage) | 独立测评OpenRouter tutorial tables | 1.106 s p50 | 59/60 intent (98.3%) | $0.0921 / 1k (author table) |
| Claude Opus (OpenRouter triage) | 独立测评OpenRouter tutorial tables | 1.957 s p50 | 60/60 intent (100%) | $2.88 / 1k (author table) |
| TypeSafe workflow Pareto claim | 厂商公布TypeSafe announcement | Vendor: ~70–500 ms class | Workflow-eval Pareto (qualitative) | Homepage cites up to 444.6× cheaper vs refs — vendor caveat: likely high-end |
These rows do not prove TenthMatrix reproduced the runs. Different datasets and cardinalities are not comparable as one ranking.
D-adjacent — Agent evaluator variance (LangChain)
LangChain/LangSmith experiment comparing judges on fixed agent traces for quality score variance and does_pass agreement with a human oracle.
- cite-2026-09-20
- 20 Sep 2026
- 样本数:5 fixed runs × 100 reps (author setup)
| 系统 | Source | p50 latency (ms) | Accuracy (%) | Cost per 1k tasks (USD) |
|---|---|---|---|---|
| Jev (does_pass agreement) | 独立测评LangChain LangSmith post | n/a in post summary | 500/500 oracle match (author) | See post for stack versions |
| GPT-5.6 Terra (does_pass) | 独立测评LangChain LangSmith post | n/a in post summary | 99.8% (author) | See post |
Does not prove Jev is the best general judge across domains. Small weather-agent dataset; defaults for LLM sampling not fully locked.
B — Model routing (TenthMatrix methodology demo)
Illustrative dashboard wiring for fixed-premium vs router arms. Numbers are placeholders for UI testing — not measured API results.
- method-0.1
- 26 Sep 2026
- 样本数:demo n=120 synthetic
| 系统 | Source | p50 latency (ms) | Accuracy (%) | Cost per 1k tasks (USD) |
|---|---|---|---|---|
| Fixed premium (demo) | 方法论演示TenthMatrix methodology demo | 4200 ms (demo) | 96% (demo) | $12 / 1k (demo) |
| Intelligent router (demo) | 方法论演示TenthMatrix methodology demo | 900 ms (demo) | 94% (demo) | $3.5 / 1k (demo) |
Nothing about real Jev or LLM quality. Do not cite externally.
C — Tool safety (TenthMatrix methodology demo)
Placeholder metrics for allow/deny/escalate UI. Replace upon first workstation_measured release.
- method-0.1
- 26 Sep 2026
- 样本数:demo n=80 synthetic
| 系统 | Source | p50 latency (ms) | Accuracy (%) | Cost per 1k tasks (USD) |
|---|---|---|---|---|
| Decision gate (demo) | 方法论演示TenthMatrix methodology demo | 180 ms (demo) | 93% detection (demo) | $0.04 / 1k (demo) |
| Fast LLM gate (demo) | 方法论演示TenthMatrix methodology demo | 850 ms (demo) | 94% detection (demo) | $0.40 / 1k (demo) |
No real detection performance. Sandbox-only future runs.