In this article 8 sections

An extraction system can score well field by field and still fail the document that matters. LlamaIndex’s new ExtractBench Pro makes that failure mode explicit. It contains 100 deterministic synthetic documents built around six patterns that customers brought to the company after its original benchmark began to saturate. The patterns include legends that change how another page must be read, internal vocabulary, organization-specific counting rules, relations drawn as lines, charts embedded in documents, and unusually long chart sheets. (LlamaIndex)

LlamaIndex tested 12 systems: four extraction APIs, five coding agents, and three tiers of its own LlamaExtract product. Each document had a 30-minute limit. A timeout or failure received zero for both Value F1 and Grounded F1. Compared with the original ExtractBench, the company reports that its own tiers lost 18.5 to 47.0 percentage points of Value F1 and 22.9 to 30.4 points of Grounded F1. Some systems completed no document perfectly.

Those are vendor-run results on a benchmark released by the same vendor. They do not establish a universal ranking. They do establish a useful evaluation design: score the completed record, not only its average fields.

Add a whole-document reliability metric

The Pro results were often bimodal. On a given document, systems clustered near perfect or far below it. An average F1 score smooths over that boundary, so LlamaIndex added percent perfect: the share of documents with 100 percent Value F1.

That metric is closer to the unit a business process consumes. A purchase order with 39 of 40 fields correct may still block payment if the missing field is the account number. A clinical form can be unusable when the extracted value is right but its page citation is wrong. A planning system may accept a plausible total that violated a counting rule defined 20 pages earlier.

Percent perfect is not sufficient on its own. A team also needs to know which fields failed, how severe the failure was, whether the value can be grounded, and whether a timeout or tool error caused the miss. Use a layered scorecard:

  1. Document outcome: perfect, reviewable, rejected, or timed out.
  2. Field outcome: value accuracy, completeness, and grounding.
  3. Failure class: convention, vocabulary, calculation, relation, chart, or geometry.
  4. Operating cost: elapsed time, model calls, tool calls, tokens, and cost per accepted document.
  5. Business loss: manual review minutes, delayed work, or the cost of an undetected error.

A promotion gate can then require both an aggregate floor and a worst-class floor. For example, a model should not pass because common invoices overwhelm a small set of failed engineering drawings in the average.

Turn production failures into a versioned suite

ExtractBench Pro’s documents are synthetic, but they are not described as unconstrained model generations. LlamaIndex says deterministic pipelines combine a configuration and seed with synthetic content, templates, and transformations. The same pipeline produces the document, extraction schema, and exact ground truth. That design anonymizes customer patterns while preserving cross-page dependencies and difficult layouts.

The original ExtractBench paper provides useful context. Its earlier suite contained 370 documents, 4,869 pages, eight domains, and 67 document types. It separated value accuracy, completeness, grounding, and cost. Pro extends that design toward the long tail rather than replacing it. (ExtractBench paper)

A local benchmark should follow a similar lifecycle:

  1. Capture a production failure as a class, not as a customer document.
  2. Reproduce the class with a deterministic generator and an explicit schema.
  3. Validate that the generated values, rendered pages, and ground-truth references agree.
  4. Run the old and candidate extractors with fixed timeout and retry policies.
  5. Keep the seed, generator version, model version, prompt, tools, and raw output with the result.
  6. Promote the case into the regression suite only after a reviewer confirms that it tests the intended failure.

The generator needs its own tests. A benchmark can reward the wrong behavior if its labels are inconsistent or if a transformation makes a target unreadable to people. Sample every generator release for semantic validity and inspect the distribution of page count, field count, layout, domain, and challenge severity. Determinism makes a case reproducible; it does not make the case representative.

Keep value and grounding as separate contracts

Pro also highlights a gap between extracting a correct value and citing where it came from. LlamaIndex reports that some coding agents and extraction APIs lost much more on Grounded F1 than on Value F1. That distinction matters when the downstream workflow needs audit evidence.

Treat grounding as a typed output rather than a prose explanation. Store a stable document identifier, page or region, quoted source span where appropriate, transformation rule, and confidence for each value. Revalidate the reference after document normalization or page splitting. If a value is derived, record both the source operands and the rule.

The benchmark’s cost plot is also a warning about hidden retries. LlamaIndex reports that coding agents reached the highest Value and Grounded F1, but at much higher per-page cost. The article cites 41 cents per page for Opus and 95 cents for Astra, while its own tiers sit on the claimed low-cost frontier below 25 cents per page. Claude often exceeded the 30-minute limit. These figures come from LlamaIndex and should be reproduced on local documents before they inform a purchasing decision.

Application watch: promote a skill only when it owns a bounded outcome

Fin published a production account of turning a Claude skill into an AI SRE agent for dead-letter queues, or DLQs. The system covers roughly 1,400 queues, 600 worker classes, and three regions. Fin says about 50 new DLQ alarms arrive each week and that DLQs had generated 80 percent of secondary on-call pages. Fin reports that the share of weekly DLQ fixes completed by the agent rose from 4 percent in the first reported week to 41 percent in week five, measured by merged pull requests. These are vendor-reported early results. The post does not provide the weekly denominator, independent validation, false-redrive rate, or longer-term defect and rollback data. (Fin)

The first version is the more useful evidence. It ran a long investigation and produced a report that operators could not verify. Fin says the model made plausible mistakes, missed critical details, and tried to hold too much context. The repair was architectural. A deterministic control plane now resolves queue identity, alarms, deployment metadata, ownership, and remediation eligibility. Isolated model runs investigate the code and choose between two bounded judgments. Typed artifacts carry evidence between stages, and a Ruby gate applies repository rules before any pull request can be proposed.

Fin also converted production incidents into a replay corpus. Sixty-three alarm fires collapsed to 33 distinct scenario shapes after duplicate patterns were removed. Code investigations were replayed against the repository state that existed before the human fix. This resembles ExtractBench Pro’s move from individual customer documents to versioned failure classes: the unit of learning is a reproducible failure shape, not an anecdote.

A skill should become an agent only when the team can name its trigger, bounded outcome, action authority, and test corpus. For a DLQ system, measure merged-pull-request survival, time to repair, recurrence, false redrives, and residual on-call work. If evidence is incomplete or policy denies action, the correct terminal state is a typed handoff to a person, not a longer speculative report.

Model watch: Timestral needs a leakage audit

Mistral released Timestral on October 15, a family of probabilistic time-series models with 5 million, 28 million, 123 million, and 209 million parameters. All four accept multivariate targets plus past-only and known-future covariates, use a context of 32,768 time steps, and emit 21 quantiles. Mistral says they run on CPUs; its published example ranges from 10 ms for the 5 million parameter model to 140 ms for the 209 million parameter model when forecasting one day from three weeks of 15-minute history on eight CPU threads. (Mistral)

The architecture separates horizon forecasting from input length and uses covariate-aware attention intended to prevent access to future target values while allowing known future inputs. Training starts on synthetic data with an 8,192-step context, then continues on a real and synthetic mix at four times the context. The loss is pinball loss over 21 quantiles.

Mistral claims leading results across GIFT-Eval, TIME, and fev-bench, but says the full technical report will follow. Until the report and submissions are available, treat the ranking as a provider claim. A production trial should use rolling-origin backtests, a naive seasonal baseline, and only covariates that would truly be known at prediction time. Check interval coverage by horizon and segment, not only mean error. The first red-team question is whether any feature pipeline reconstructs future information that the attention mask was designed to exclude.

API watch: voice decisions move inside the session

OpenAI’s October 15 API changelog adds native Decisions integration to GPT-Live. A live session can delegate a typed decision directly or trigger decisions from input and output transcript deltas. (OpenAI changelog)

The implementation guide makes the authority boundary clear. Decisions selects among declared choices, while the application performs the action, verifies the observed state, and reports the result back to the voice session. Native delegation removes an application round trip, but it should not remove application-side authorization, idempotency, or audit records. (OpenAI voice guide)

For a state-changing action, record the transcript item, decision schema and answer, policy check, attempted effect, observed effect, and response to the user. A low-latency classifier is not a transaction coordinator.

Stay Sharp: test the generator with properties

QuickCheck introduced a durable testing pattern in 2000: state properties of a program, generate many inputs, and report counterexamples that violate those properties. It also supports custom generators for domain-specific data. (Claessen and Hughes)

Apply that idea to synthetic evaluation pipelines. A generated extraction case should satisfy properties before it reaches a model:

  • every ground-truth field maps to the declared schema;
  • every grounded value points to a page or region that contains its evidence;
  • totals equal the values produced by the declared counting rule;
  • changing a legend changes only the fields governed by that legend;
  • serializing and reloading the configuration preserves the same document and labels.

When a property fails, reduce the case to the smallest document, page set, or configuration that still violates it. A small counterexample is easier to diagnose than a 90-page synthetic report. Keep passing seeds only if their distribution covers the failure classes you intended. Random generation explores a space; it does not prove that the space matches production.

Watch next

  • Independent runs of ExtractBench Pro, especially percent-perfect results by failure class and the effect of the 30-minute cap.
  • Whether document extraction vendors publish grounding errors separately from value errors.
  • Timestral’s technical report, benchmark submissions, and calibration results on shifted production data.
  • Failure and latency evidence from GPT-Live decisions when actions require authorization or must be idempotent.