In this article 6 sections

Meta has put an AI test generator into its production review path and says it helped catch 74 serious privacy bugs before code landed in the first half of 2026. The system does not ask a model to review a diff and explain whether it looks risky. It generates an executable test that passes on the parent revision and fails on the proposed change. (Meta Engineering)

The company reports that 589 engineers, 64% of those contacted, engaged with the findings. Those engineers labeled 60% of the surfaced findings as true positives. Meta also reports a 20% reduction in privacy severity incidents during the same half-year period. The account does not publish the total number of targeted diffs, the number of findings dismissed without engagement, or a comparison group. The 20% reduction is therefore an operating result associated with the deployment, not a causal estimate of the test system’s effect.

The useful advance is narrower and more portable. A generated explanation asks a reviewer to trust another interpretation of the code. A generated counterexample gives the reviewer a behavior to reproduce.

Generate a disagreement, then prove where it came from

Meta runs two workflows overnight on code changes selected as high risk. The intent-aware workflow infers the purpose and risks of a diff, creates mutants that represent plausible implementation mistakes, then generates tests that distinguish those mutants from the parent. The dodgy-diff workflow makes no claim about intent. It treats the proposed change itself as the mutant and tries to expose a behavioral difference.

A candidate becomes a weak catch when its test passes on the parent and fails on the diff. That is evidence of changed behavior, not yet evidence of a bug. An intended behavior change will produce the same pattern.

Meta ranks weak catches with an ensemble of model judges and a deterministic rule-based assessor called RubFake. Engineers are contacted only for candidates above the resulting threshold. The production design keeps the human responsible for the final oracle: was the observed change intended, and is the intended change acceptable?

The earlier FSE paper provides a smaller, more inspectable baseline for the approach. Across 22,126 generated tests, diff-aware methods produced weak catches for 4.0% and 7.9% of targeted diffs, depending on the workflow. During the paper’s initial deployment, 41 engineer contacts produced eight confirmed bugs, four of which Meta classified as severe. The paper reports that model and rule-based assessors reduced manual review load by about 70%, although their agreement was only modest. (FSE paper)

That modest agreement is useful information. Two assessors can be complementary without either becoming ground truth. Preserve the evidence chain for every promoted finding:

  1. Parent and diff commit digests.
  2. Generator, prompt, model, and toolchain versions.
  3. The generated test and any mutant that motivated it.
  4. Parent and diff execution results from the same environment.
  5. Every assessor score and filtering rule.
  6. The engineer’s disposition, reason, and eventual production outcome.

Without that chain, a team can count accepted findings but cannot replay why a candidate was selected, measure policy drift, or determine whether a new judge would have hidden an old catch.

The oracle needs a policy layer

Intent inference is useful for targeting. It is not a safety specification. A change can match its author’s intent and still violate a privacy invariant, an authorization rule, or a data-retention policy. The human question should therefore have two parts: was the behavior intended, and is it permitted?

Encode stable invariants outside the generator. A privacy test can assert that one tenant never observes another tenant’s record. An authorization test can verify that a write fails without the required actor and scope. A schema migration test can require backward-readable data through a declared rollback window. The generated test searches for a counterexample; the policy layer decides whether that counterexample blocks merge.

Start the system in shadow mode and report its selection funnel. Useful denominators include targeted diffs, generated tests, weak catches, candidates surviving each assessor, engineer contacts, engineer responses, confirmed defects, severe defects, and escaped incidents. Add median review time and the age of unresolved findings. Precision alone can be improved by contacting almost nobody.

The release gate should also test the testing system. Replay confirmed catches and representative false positives whenever the generator, model, rubric, execution environment, or threshold changes. A judge upgrade that reduces review load but loses a known severe catch is a regression.

Architecture watch: discovery is not exposure

A practitioner synthesis published on October 28 resurfaced Uber’s October 1 account of its MCP Gateway. The underlying report is older learning, but its control-plane decisions remain unusually concrete. Uber says the gateway fronts more than 800 MCP servers and 5,000 tools across HTTP, gRPC, TChannel, native MCP, and third-party services. (Uber Engineering)

AutoCrawler scans Uber’s interface registry and creates or updates tool definitions. Crucially, every discovered server and tool starts disabled. The owning service team must review the generated definition, approve its configuration diff, and enable it. The runtime then applies actor-aware authorization at server and tool granularity and redacts sensitive response data.

That separates four decisions that small deployments often collapse:

  • Discovery: an API exists and can be represented as a tool.
  • Publication: an owner approves its name, schema, and description.
  • Authorization: this actor may invoke this tool for this purpose.
  • Execution: this request passed current policy and reached the service.

Uber also uses incremental discovery and response projection so an agent does not load thousands of schemas or entire payloads into context. That reduces token pressure, but it makes the discovery path part of the authority path. Log which candidate servers were considered, which schema version was selected, which fields were projected away, and which actor identity reached the backend.

A central gateway concentrates risk. Treat configuration rollout, identity propagation, redaction, rate limiting, and audit delivery as independent failure domains. Version every tool contract, fail closed on an unknown actor or stale policy, and preserve a direct rollback to the last approved definition. A registry entry is not evidence that an effect is safe.

Application watch: code mode pays when work becomes a loop

Pydantic published an October 28 benchmark comparing direct tool calling with code mode, where a model writes a small program that invokes tools inside a sandbox. The study covered 5,276 graded runs across eight models, five CRUD applications, and workloads ranging from five to 200 items. Pydantic built the harness and the benchmark code is not public, but the post specifies the tested versions, arms, graders, run counts, pricing method, and known ambiguities. (Pydantic)

On 12 long-horizon tasks, Pydantic reports that code mode reduced cost by 58% and wall time by 51% across the eight models, while success increased by 16 percentage points. In the scaling tests, direct tool-calling success fell to 62% to 79% at 200 items; code mode finished at 92% to 100%. The benefit was not universal. At five items, code mode cost 21% to 38% more on three of the five tasks, and success on everyday tasks was effectively flat.

The mechanism is more specific than fewer model turns. Current models already batch large numbers of tool calls. Code mode kept intermediate records and repetitive control flow inside the sandbox, so the model did not have to re-read every item or spell out every call. In one 200-invite run, input tokens fell from 183,723 to 34,461 while model requests moved only from eight to six.

That changes the architecture boundary. The generated program can loop, branch, and fan out effects much faster than a conversational agent. It therefore needs the same controls as any execution environment: no ambient host or network access, explicit tool capabilities, execution and memory budgets, bounded concurrency, idempotency for retried writes, and per-call authorization and audit. Pydantic’s Monty setup starts without host or network access and limits each snippet to 30 seconds and a 256 MiB heap by default. Those are useful defaults, not proof that the registered tools themselves are safe.

Choose the mode from the workload. Direct tool calling remains simpler for small jobs, ambiguous exploration, and actions that benefit from stepwise human inspection. Code mode is a stronger candidate for bounded, repetitive transformations with deterministic end-state checks. Before adopting it, measure the crossover point on your own tool payloads, model mix, pricing, and failure recovery. The published result isolates bookkeeping over small applications; it does not establish the same gains for production systems with long-running effects, rate limits, or partial failures.

Stay Sharp: mutate only where the decision can change

Google’s 2021 paper Practical Mutation Testing at Scale explains why mutation testing becomes intractable when every possible mutant is generated across an entire codebase. Google’s production approach mutates only changed code during review, filters candidates unlikely to matter to developers, limits mutants per line and review, and selects mutation operators using their historical performance. The evaluation covered more than 24,000 developers and 1,000 projects. (Google Research)

The lesson for AI-generated testing is not simply to generate more tests. It is to spend generation and review budget where a result can change a merge decision.

Create a per-language and per-component ledger of mutation operators, generated candidates, valid tests, confirmed defects, severe defects, and reviewer cost. Retire operators that mostly create equivalent or irrelevant mutants. Keep a small exploration budget for new operators so the system does not optimize only for familiar bug classes.

A mutation score measures whether tests detect seeded changes. It does not measure production reliability. Pair it with escaped-defect data and with replay of real historical failures. The strongest release evidence is a three-part result: generated tests kill relevant mutants, known defects are rediscovered, and the review funnel remains affordable enough that engineers still respond.

Watch next

  • Meta’s full selection funnel, including targeted diffs, nonresponses, false negatives, review time, and severe catches per thousand targeted changes.
  • Independent replication of diff-aware generated tests outside Meta’s languages, tooling, and privacy review process.
  • Uber gateway measurements for authorization failures, schema drift, redaction misses, policy propagation, tool-selection quality, and rollback time.
  • Independent replication of code mode across production tools, partial failures, rate limits, and stateful retries.
  • Evidence that test generators preserve policy invariants when a diff’s stated intent conflicts with privacy, security, or rollback requirements.