Overview
Jev is TypeSafe's flagship System One decision model: instead of generating text, it answers typed questions (Choice, Score, Noul) about a state with calibrated probabilities that software can branch on directly. This work presents daf-jev, an open, modular, composable Python toolkit for the Jev API, together with a live-API characterization of the model. The package layers ergonomic question builders, a decision-composition library (confidence gates, tiered routing, composite scoring), a concurrent corpus evaluator, a calibration module, a CLI, an agent skill, and a Model Context Protocol (MCP) server over a thin typed client. Live benchmarks show that batching questions into one call is faster and cheaper than sequential calls (up to ~18x speedup and ~4x fewer tokens), that decision pipelines complete within the model's millisecond envelope, and that reported confidence is self-consistent across repeated evaluations. All numbers in the accompanying manuscript are generated from benchmark artifacts; the full provenance chain (hashed documentation snapshot, figure registry, validation receipts) is machine-checked.
Overview source: Curated paper metadata.
Methods and contributions
Read the source for the full argument, qualifications, and evidence.
Findings and contributions
- Batching questions into one call gave wall-time speedups of 3.98, 8.77, and 18.55 across the configured batch widths.
- At the widest batch width the sequential strategy used 4.22 times the tokens of the batched call, so batching was both faster and cheaper.
- End-to-end decision pipelines had median wall times of 0.133 s (composite scoring) and 0.129 s (intent routing).
- Under a self-consistency proxy, the calibration benchmark reported ECE 0.0730 and Brier 0.0252; the author notes this shows consistency with proxy labels, not correctness about the world.
- The unit test suite reached 93.08 measured coverage.
Methods
- daf-jev layered Python client for the System One endpoint — Implements typed builders (noul, choice, score), sync/async clients with a pure retry policy, and a typed exception hierarchy over one HTTP endpoint.
- Pure-function decision patterns and a concurrent batch Evaluator — composite_score, confidence_gate, and route/pick turn typed answers into decisions locally; the Evaluator runs a fixed question set over many states without aborting on failures.
- Live-API batching benchmark at batch widths 5, 10, and 20 — Compares one batched call against sequential single-question calls on the jev-latest model, recording wall time and token totals.
- Self-consistency calibration benchmark (6 states × 5 repeats) — Scores choice answers against the modal choice across repeats and noul answers by pairwise stability, computing ECE and Brier score without external ground truth.
- No-mock test suite and hash-manifested documentation snapshot — Unit tests drive the real transport against a local HTTP server; model claims are grounded in a 108-page hashed snapshot of the vendor documentation.
Key concepts
- paired publication
Summary sources: Paper metadata and evidence · Learning guide · Extracted source text.
PDF downloads
Archived files available directly from this site.
- daf-jev_combined.pdf PDF · 0.71 MiB · extracted-text source
Citation
Citation metadata follows the unified bibliography.
Catalog details and resources
Related in Computational
Other catalogued works in the same domain.