A typed evaluator matched five labels across 500 repeated decisions

LangChain ran the same five captured weather-agent traces through four evaluators 100 times each. A human reviewer supplied the pass/fail labels. Jev, a new model built to return typed decisions rather than prose, agreed with those labels on all 500 repeated decisions. GPT-5.6 Terra agreed on 99.8%, GPT-5.6 Luna on 96.4%, and Claude Sonnet 4.6 on 80.0%. Jev also averaged 0.44 seconds and $0.00035 per call. (LangChain)

That is a useful result, but it is not a general ranking of evaluators. Five fixed weather traces cannot establish performance across ambiguous support conversations, long agent trajectories, safety decisions, or shifting production data. What the experiment does establish is that evaluator architecture now has another credible option. Teams no longer have to choose only between deterministic code and a generative model asked to emit a score.

The test separates correctness from repeatability

LangChain measured two different properties. Its binary metric asked whether each judge agreed with the human label. Its continuous metric asked whether repeated judgments of the same trace produced the same score.

Jev’s mean per-case variance was 0.0000149. LangChain reports that Luna’s variance was 433 times higher, Terra’s was 913 times higher, and Claude’s was 92 times higher. That repeatability matters for regression testing. If an unchanged trace receives a materially different score on each run, a release gate can move even when the system under test did not.

Repeatability is not correctness. A judge that returns the same wrong answer every time has perfect repeatability and no decision value. This experiment also used one human reviewer as the oracle, did not record the Jev service version, and left the generative judges on provider defaults. LangChain publishes those limitations and calls the result early. They should remain part of any adoption decision, not disappear behind the headline numbers.

A decision model can sit between rules and generative judges

TypeSafe describes Jev as a “System One” model. It accepts text or structured state, evaluates predefined questions, and returns choices, ordered scores, or Boolean probabilities. It does not generate explanations. TypeSafe says its training objective targets calibrated decisions and that its output schema cannot contain type errors. Those are provider claims, and the company acknowledges that some published comparisons favor its model and that early access remains limited. (TypeSafe AI)

The useful architectural idea is narrower than the branding. A production evaluation cascade can use four layers:

1. Deterministic checks handle facts that code can prove, such as schema validity, tool completion, permission boundaries, and exact identifiers.
2. A typed probabilistic evaluator handles bounded questions whose answer set is known in advance, such as routing, rubric levels, or whether a trace needs review.
3. A generative judge handles cases that require an explanation, synthesis across evidence, or an open-ended critique.
4. A person owns ambiguous, high-impact, or audited decisions.

Vercel’s new experimental evaluation API exposes this shape directly. Application code receives probabilities, applies its own thresholds, routes clear cases, and sends uncertain cases to review. The model assesses the state; the application still owns the consequence. (Vercel)

That separation is more important than the model choice. It keeps policy in versioned code, makes thresholds testable, and allows the evaluator to be replaced without rewriting the workflow. It also prevents a confidence score from silently becoming authority.

Typed output does not make the judgment safe

Langfuse’s integration guide identifies three limits that should shape a rollout. Jev cannot abstain natively, it provides no rationale, and its accuracy can decline when long state contains irrelevant material. A forced binary question will still produce an answer when neither option is justified. A low-cost call can therefore scale a systematic mistake as efficiently as it scales a useful signal. (Langfuse)

Add an explicit escape path in the application even when the model does not provide one. For a binary decision, define a middle probability band that routes to another evaluator or a person. For a multi-class decision, consider both the winning probability and the gap to the runner-up. Do not treat a returned confidence field as calibrated until labeled production examples show that it is.

Version the entire decision contract: model version, question text, criteria, input projection, threshold, and downstream action. Keep the raw trace and the evaluator output together. If the model version or question changes, replay a fixed labeled set before comparing new scores with the old series.

Long traces need an input policy too. Pass the minimum state required for the decision, and test whether removing apparently irrelevant spans changes the verdict. If it does, the evaluator is reading a different task from the one the team thinks it specified.

Stay Sharp: calibrate the action, not only the probability

A model is calibrated when events assigned probability 0.8 occur about 80% of the time across comparable cases. Production decisions need one more step: the threshold should reflect the cost of each error and the cost of review.

Suppose an evaluator estimates a 20% probability that an agent response caused a billing error. That number is not inherently high or low. If a missed error costs far more than a manual check, 20% may already justify escalation. If the action is reversible and review is expensive, the threshold can be higher.

Build reliability tables by task class and consequence, not only one aggregate curve. A judge may be well calibrated for ordinary routing and overconfident on rare security cases. Monitor the share of cases entering the uncertain band, human disagreement by class, and performance after input or policy changes. Calibration is conditional on the data and decision contract that produced it.

What to watch

• LangChain’s September 22 technical session is a concrete chance to clarify model versioning, broader task coverage, and the failure cases behind the aggregate result.
• Independent tests on larger, imbalanced datasets will determine whether the weather-agent result survives rare failures, long traces, and disagreement between multiple human reviewers.