In this article 4 sections

An early OpenAI exploit run produced 200 successful attempts against assigned Linux kernel vulnerabilities. Twenty reached root through a different zero-day. The agent achieved the terminal state while failing the causal task that operators thought they had measured.

That result is more consequential than the headline vulnerability count. It shows that an outcome-based grader can certify the wrong mechanism, even when the outcome is technically real. OpenAI responded by adding an independent check that the assigned vulnerability lay on the exploit path. (OpenAI)

The lesson applies beyond offensive security. When an agent can choose among many paths to a valid state, success needs both an end-state assertion and evidence about how the state was reached.

Treat the causal path as part of the acceptance test

OpenAI’s full pipeline scanned 5,034 audit targets covering 2.8 million lines of code. It produced 2,308 deduplicated candidate findings, then dynamically reproduced 1,724 of them. Exploit agents later consolidated their work into 948 distinct local privilege escalations. OpenAI reports an API cost of $1.21 million for the full run, or $1,279 per distinct escalation.

These are provider-reported results, not an independent benchmark. The run also used stock desktop protections rather than every kernel hardening profile, and some novelty judgments relied on expert review of a sample. Still, the pipeline exposes four controls that a production agent system can adopt now:

  1. Verify the assigned mechanism. Record which input, code path, tool call, or state transition caused the accepted outcome. A shell, completed ticket, or valid JSON object is insufficient when multiple hidden routes can produce it.
  2. Revalidate against current state. During the ten-day run, 61 issues were patched, 43 had fixes staged, and 299 entered public discussion. Evidence tied to an older commit is not evidence about the current build.
  3. Price failed search separately. OpenAI reports an average API cost of $360 for successful exploit development and $3,922 for unsuccessful attempts. A stopping rule should consider marginal evidence gained, not only an overall token budget.
  4. Remove orchestration that no longer earns its cost. A 103-technique planning layer lifted an earlier model’s success rate from 76.2% to 88.9% on 63 candidates. With a later model, it added only 1.5 percentage points, so the team removed the heavier layer.

The last point prevents an old optimization from becoming permanent ceremony. Model capability can change which scaffolding is useful. Keep the ablation test beside the orchestration code, and rerun it when the model, sandbox, or task distribution changes.

For a defensive workflow, translate the same design into a signed evidence bundle: target commit, assigned finding, reproduction trace, exploit path, mitigation profile, patch status, and independent path check. The agent can propose severity and remediation, but a release gate should consume the bundle rather than a single success flag.

A small extractor can win a narrow contract without becoming a general document system

Dharma AI released a 2B parameter model for schema-directed field extraction. The model takes a document and requested keys, returns exact key names, and emits null when a field is absent. Dharma says it trained a LoRA on 43,384 examples and evaluated 546 documents across three scenarios, yielding 1,638 cases. (Dharma AI)

On Dharma’s benchmark, its model scored 0.834, compared with 0.816 for Claude Opus 5 and 0.741 for GPT-5.6 Luna. The metric combines field-path matching, including table position, Levenshtein similarity, and valid JSON rate. The team also estimates a sixfold to 147-fold cost advantage when running on one L40S GPU rather than paying list API prices.

The practical candidate is a bounded service, not a universal replacement. The comparison is vendor-run, the complete benchmark has not been independently reproduced, and cost depends on utilization, batching, operations, and error handling. A normalized string match also cannot establish that a value came from the correct evidence span.

A responsible evaluation should therefore separate five invariants:

  • schema validity and exact field names;
  • semantic correctness after type normalization;
  • evidence provenance down to page, region, or source span;
  • calibrated abstention when the document is ambiguous;
  • downstream business impact, including the cost of a confident wrong extraction.

Start with a shadow deployment on one document family. Version the schema, preprocessing, model weights, and normalization rules together. Route low-confidence or conflicting fields to review, and measure correction work per accepted document. If a 2B model wins that contract, it has created a useful system boundary. It has not proved that every document workflow should share the same model.

Architecture radar: Kimi’s weights arrive with operational conditions

Moonshot AI’s Kimi K3.1 Pro weights are now available on Hugging Face. The model card describes a 2.8 trillion parameter mixture-of-experts model with 104 billion active parameters, 93 layers, multimodal input, and a 1,048,576-token context window. The repository contains 96 weight shards and is about 1.56 TB. (Moonshot AI)

Call it open-weight, not open source. The custom license permits broad use and modification, but a model-as-a-service operator with more than $20 million in aggregate revenue over any consecutive 12 months must obtain a separate agreement before commercial use. Products over 100 million monthly active users or $20 million in monthly revenue must display the model name, subject to listed exceptions. (Kimi K3.1 Pro License)

Security behavior is another versioned property. On October 24, Reuters reported Moonshot’s statement that training changes reduced K3.1 Pro’s score on an internal cyber benchmark from 90.5% to 81.0%. That result has not been independently replicated. (Reuters) Teams evaluating the downloadable release should pin the weight commit, inference code, quantization, license version, safety policy, and cyber test set. “Kimi K3.1 Pro” alone is not a reproducible deployment identity.

Stay Sharp: metamorphic testing finds failures without a perfect oracle

Many agent tasks lack a cheap answer key. Metamorphic testing addresses that oracle problem by checking relations between executions instead of requiring the exact output for every input. NIST describes the method as generating follow-up tests whose results should preserve or predictably change a known property. (NIST)

For document extraction, reorder requested fields. The normalized values should remain the same. Add an optional key that is absent from the document. Existing fields should remain unchanged and the new key should be null. For a vulnerability reproducer, apply the patch for the assigned bug. The exploit should fail for the expected reason while unrelated regression tests continue to pass.

The relation must be valid for the domain. Reordering pages may change meaning, and patching one bug may expose another path. Metamorphic tests do not replace direct assertions or provenance. They add a second route to evidence when the final state is easy to produce and the intended mechanism is hard to observe.

The operational decision is to stop accepting terminal success as a complete proof. Bind success to the target version, causal path, and invariants that the system is supposed to preserve. That turns an impressive result into one that can survive a release gate.