Three systems ran on the exact same 300 AG News test samples (4 classes: world, sports, business, sci_tech), one identical question, one 36-core CPU box with no GPU. This task is out of distribution for all three, so the numbers describe generalisation, not memorisation.
Results
- laya (ModernBERT-large 421M, local): accuracy 0.9533 · ECE 0.2150 · mean confidence 0.739 · 236 ms/sample · 0 tokens
- open-jev-deberta-v3-large (DeBERTa-v3-large, local): 0.7167 · ECE 0.1032 · 304 ms/sample
- Jev API (jev-1.13.0): 0.8400 · ECE 0.1012 · confidence 0.935 · 521 ms/call · 417 tokens/call · 0 errors in 300 calls
The number that decides is coverage at the gate threshold
These models earn their keep by using confidence to decide what stays local and what escalates to an LLM. At confidence ≥ 0.7: laya covers 67% of traffic at 99.5% accuracy; the DeBERTa build covers 38% at 84.3% (only 3 of 300 samples clear 0.85). Jev covers 85% of traffic at ≥ 0.85 with 89.8% accuracy. For maximum token savings laya wins; for handling the most work locally Jev wins.
Guardrails show the widest gap
On the same 12 harmful and 12 benign prompts: Jev scores harm 3.00 against 0.03, catches 12/12, and raises no false alarm. laya scores 1.70 against 0.99, only 3/12 clear the threshold, and its jailbreak flag fires 9/12 with one false positive. As a pre-action check, that gap matters more than classification accuracy.
Recommendations
- Moderation, jailbreak resistance, checking an action before it executes: use the Jev API — $0.042 per million input tokens, output free, about $0.018 per 1,000 calls.
- High-volume classification that needs accuracy and costs no tokens: run laya locally.
- Self-hosting with good calibration: the open DeBERTa build, accepting lower accuracy.
- Do not use any of them for arithmetic, threshold comparisons or position sizing — see The hidden value is when you get the data.
Sources: laya on Hugging Face, TypeSafe's Jev docs, open-jev-deberta-v3-large, typed-decisions repo.