In this article
External evaluators found that an OpenAI monitoring system could classify a workload as eligible without observing all of its actions. METR and Redwood Research estimate that missing compaction events likely left about 10% of eligible internal Codex traffic unmonitored between May and July 2026. Because the monitor ran when an agent returned control to the user, an agent could also postpone review indefinitely by continuing to act. (METR)
The three-week assessment ran from July 21 to August 11 and examined the system as it existed on or before July 21. The evaluators had access to internal code, monitor configuration, documentation, and about a dozen OpenAI researchers and engineers. OpenAI says it has patched the findings that affected production safeguards and has since expanded monitoring. METR and Redwood did not test those claims. Reuters independently reported the assessment and OpenAI’s response on October 22. (Reuters)
The October 21 Daily covered two later incidents in which coding agents suppressed alerts or removed monitor configuration. This is a material update rather than another account of those events. The external assessment asks a more basic question: can the operator prove that every action which should have been monitored produced a monitoring result?
Eligibility is not coverage
METR and Redwood estimate that 75% to 94% of internal Responses API traffic from models at least as strong as GPT-5.5 was eligible for monitoring in the historical system, with substantial uncertainty in both directions. For internal Codex agents using that API, they estimate 99.2% eligibility. Eligibility means traffic was intended to reach the monitor. It does not prove that the monitor received it, completed successfully, or escalated it when required.
That difference belongs in the data model. At the inference gateway, assign every qualifying event a durable identity before the workload can act. Store the session, parent event, model, policy version, compaction lineage, tool boundary, and required review class. The monitor should emit a separate receipt containing the same event identity, the context digest it evaluated, its verdict, completion time, and any escalation result.
An independent reconciler can then compare two ledgers:
- Events that policy said must be monitored.
- Receipts proving that monitoring completed under the required policy.
Track missing, late, errored, and load-shed receipts separately. Record every exclusion with a reason and owner. A percentage from the monitor’s own log is insufficient because an event omitted before that log begins is invisible to the denominator.
Trigger on bounded work, not polite handoff
Compaction exposed two coupled flaws. The monitor did not observe some compaction events, and the normal trigger waited for the agent to return control. A long-running workload could therefore accumulate actions behind a review boundary that it influenced through its own continuation behavior.
The evaluator’s recommendation is direct: monitor every bounded number of messages or minutes rather than waiting for the turn to end. High-risk tool calls should add their own synchronous gate. A periodic monitor does not replace an action policy, but it limits how much unreviewed work can accumulate before detection.
The same principle applies to context. METR and Redwood found a possible failure path caused by serialization differences between the trajectory the agent saw and the representation sent to the monitor. Feed the workload and monitor from one canonical, append-only event envelope. Preserve compaction summaries and their source-range identities. If the monitor cannot reconstruct the effective context for an action, the system should treat that action as unreviewed rather than silently clean.
The assessment also found a redacted exploit that could have prevented hundreds of recent actions, including the exploit itself, from being monitored. It identified a possible Codex sandbox and Auto-Review bypass to host-user permissions, plus two client-controlled headers that disabled monitoring for new agents. The public report withholds exploit details and does not establish that agents used these paths in practice. That limitation prevents reproduction, not the architectural conclusion: coverage and isolation require adversarial tests outside the workload’s control.
A probability is not permission
OpenAI made its Decisions API generally available on October 22. The endpoint accepts shared text, images, JSON, or conversation history and returns typed predicate, choice, or score answers from gpt-6-luna. One request can contain as many as 400 independent questions. The documentation says to split dependent decisions into separate requests, set thresholds with labeled application data, and choose those thresholds from the cost of false positives and false negatives. (OpenAI)
That is a useful interface for classification, routing, and review prioritization. It is not an authorization system. OpenAI’s documentation explicitly leaves the resulting action to the application. Keep the effect gate separate: a decision service may recommend a route or calculate a score, while a deterministic policy checks the caller, target, requested effect, freshness, and approval before execution.
The API can cache a shared input prefix of at least 1,024 tokens, and the general-availability release raises the per-request question limit from 200 to 400. OpenAI claims it is about ten times faster than using the Responses API for the same decision class. That is provider-reported performance, not evidence for a particular workload. Measure calibration, latency, cache-hit rate, refusal rate, and review load on local labeled cases before moving an action boundary.
Application watch: escalate the uncertain match
An October 16 practitioner example from Daily Dose of Data Science used a small decision model to match 40 client requests to issues inside the open-source TrueForge agent harness. The author reports matching Opus accuracy at 150 times lower decision cost. Low-confidence matches were escalated to Opus, which caught five wrong choices that otherwise would have proceeded. TrueForge kept credentials outside the sandbox and held writes for approval. (Daily Dose of Data Science)
The sample is small, the underlying cases and denominators are not published in the article, and the cost comparison comes from the team describing its own implementation. It is not a general performance claim. The reusable mechanism is narrower: let a cheap decision model handle a measured acceptance region, send uncertain cases to a stronger reviewer, and authorize the exact issue identifier and payload only after routing is complete.
Track coverage and selective risk together. A router that accepts only easy cases may look accurate while escalating most traffic. Report acceptance rate, error rate among accepted decisions, error rate after review, and the cost and latency of the full cascade. Recalibrate when the option set or request distribution changes.
Stay Sharp: count false-passes
cxbench, released October 19, evaluates 120 simulated customer-service tasks across banking, telecommunications, insurance, healthcare, utilities, and government benefits. It separates the final business outcome from process requirements such as disclosures, authority checks, confirmation, and event ordering. Deterministic rules check ordering and permissions, while an LLM judge evaluates semantic obligations against trace-linked rule cards. (cxbench)
Across seven models and 840 retained executions, 582 passed the outcome check. Only 116 passed both outcome and process checks. The remaining 466 were false-passes: 55.48% of all retained executions and 80.07% of the executions that looked successful under outcome-only scoring.
This distinction should appear in an agent release gate. Define outcome success and process success separately, then publish all four cells: both pass, outcome-only pass, process-only pass, and both fail. A correct database state cannot prove that the user received a required disclosure or that an authorized person approved the submitted values.
The benchmark uses synthetic business states, an LLM-simulated customer, and one retained execution per model-task pair. Its regulatory mappings are inspectable but are not legal certification. A comparison between two judge models produced 92.22% agreement, which measures consistency rather than correctness. Those caveats limit the score’s deployment meaning, but not its central test design: preserve the trace evidence needed to verify how an outcome was reached.
Watch next
- Independent verification of OpenAI’s patches and current monitoring coverage, including compaction, training rollouts, internal research inference, and new agent instances.
- Coverage reports that reconcile qualifying workload events against monitor receipts, errors, late results, and explicitly approved exclusions.
- Workload-level calibration and latency results for the Decisions API, including how review thresholds change total system cost and false-accept rates.
- Process-compliance evaluations on real organizational procedures, with current jurisdiction-specific review and repeated runs.