In this article
The Reserve Bank of Australia’s business liaison now finds AI use almost universal among the firms it hears from. Adoption depth is a different picture. Roughly 60% report limited use, about 30% moderate use, and around 10% deep or transformative use. The productivity distribution lags further: about 35% report no gain, 30% a minor gain, 23% a moderate gain, and 11% a significant gain. (Reserve Bank of Australia)
Those figures are evidence from the RBA’s liaison contacts, not a representative census or a causal estimate. They still expose an architecture problem that model adoption dashboards miss. A team can add copilots, gateways, retrieval, and agents while leaving the workflow, decision rights, and outcome measurement unchanged. Usage rises immediately. Productivity may not.
Two other current items make that distinction operational. OpenAI’s BioBasicsBench evaluates whether agents can interpret raw laboratory evidence rather than polished paper figures. CISA’s November 10 deadline for an exploited Brotli decoder flaw shows why platform inventories must follow executable behavior, not only declared packages. Both require teams to define the evidence that permits an action before an AI system or service reaches production.
Measure the redesigned workflow, not the model rollout
The RBA describes the familiar productivity J-curve: organizations spend first on experimentation, infrastructure, training, and process redesign, while measured productivity stays flat or falls. Gains arrive only if complementary changes take hold. Electrifying a factory did not create its largest benefit until the factory itself was reorganized. The same warning applies to AI.
For an architecture portfolio, the unit of value should therefore be a completed workflow with an observable outcome. Model calls, active seats, accepted suggestions, and agent runs are operating signals. They are not the benefit case.
A practical baseline has four layers:
- Work unit: define the business event that starts and ends the workflow, such as resolving a support case, reviewing a contract, or shipping a verified change.
- Outcome: measure quality, elapsed time, rework, incidents, and the amount of expert review needed for that work unit.
- Complementary change: record which policy, data contract, approval boundary, interface, or team responsibility changed alongside the model.
- Counterfactual: compare against the prior workflow or a concurrent control, not against a vendor benchmark.
This structure separates three failure modes. A capable model can be trapped in an unchanged process. A redesigned process can move work faster while exporting review effort to another team. A local task can improve while the end-to-end workflow accumulates queues, retries, or compliance checks. Portfolio reviews need all three views.
The RBA also reports that about three-quarters of liaison contacts saw no headcount change from AI over the past year. Roughly 12% reported slower hiring or unfilled roles, around 8% lower headcount, and about 6% higher headcount. The Bank explicitly treats the labor-market differences as tentative and potentially confounded. That is the right measurement posture. Exposure, use, task speed, workflow output, and employment are different variables. An architecture review should not collapse them into one adoption score.
BioBasicsBench tests whether an agent respects the evidence
OpenAI and Ligo built BioBasicsBench from 100 routine wet-lab tasks across protein methods, imaging and cytometry, assays, nucleic acid methods, and molecular cloning. Inputs include gels, microscopy stacks, instrument exports, sequencing files, handwritten notes, protocol videos, and photographs of apparatus and plates. Scientists wrote the tasks from their own experiments, then each task passed an independent internal review, an external review by a working PhD-level scientist, and a final internal review. (OpenAI)
The evaluation ran seven model families at 37 model and reasoning-effort settings. Each setting received five attempts on every task, producing 18,500 results. At xhigh effort, GPT-6 Astra earned 66.7% mean reward, compared with 43.8% for GPT-5.6 Sol. Whole-task bootstrap intervals put Astra between 60.3% and 73.1%. Astra also used fewer output tokens per task at that setting, 4,704 versus 9,099.
The failure examples matter more than the rank. Astra noticed an incorrect antibiotic label that invalidated a control, but in another task it attached the wrong experimental meaning to correctly separated PCR bands. It inspected lighter and darker vacuoles, then discarded the observations that contradicted its preferred conclusion. On a colony-counting task, its net count was about 12% high because it merged neighbors, missed visible colonies, and marked empty agar.
That is not a generic vision problem. It is an evidence contract problem. A lab assistant must preserve which observation supports which conclusion, identify when a control is invalid, and stop when the evidence cannot decide. A high aggregate reward does not show that any one of those behaviors is reliable enough for autonomous action.
For deployment, split the benchmark into operational gates:
- evidence extraction, where every measurement points back to a source region or raw record;
- control validation, which can invalidate the downstream conclusion even when the primary image looks clear;
- contradiction handling, which must retain observations that oppose the leading hypothesis;
- action policy, which maps uncertainty and error cost to recommend, ask, abstain, or execute.
Then evaluate the complete system on local instruments, protocols, labels, and failure costs. The benchmark is provider-authored and developed with one biotech partner. It establishes a useful task design and a bounded provider result, not readiness for a particular laboratory.
Deadline watch: trace Brotli to the decoder that will actually run
CISA added CVE-2026-111497 to its Known Exploited Vulnerabilities catalog and set November 10 as the federal remediation date. The flaw is an out-of-bounds write in the Brotli decoder. A crafted compressed stream can corrupt adjacent process memory, disclose data, crash the process, or potentially execute code. The CVE record covers Brotli 1.0.3 through 1.2.0, with fixed releases on the maintained branches. (CISA) (CVE)
The difficult part is inventory. Brotli may be bundled into a runtime, linked into a self-contained binary, shipped in a container base image, or installed as a language package. Node.js lists affected 22.x, 24.x, and 26.x release lines and shipped fixes in its October 21 security releases. (Node.js) Microsoft says affected .NET applications must update the runtime or SDK, then rebuild self-contained and NativeAOT deployments because replacing the shared runtime does not replace a decoder already linked into an artifact. (Microsoft)
Presence alone is not exposure. The dangerous path processes attacker-controlled Brotli data. Inventory should therefore join four facts:
- the decoder implementation and version in the deployed artifact;
- whether the service accepts or fetches untrusted compressed content;
- which ingress, client, proxy, tool, or document pipeline performs decompression;
- whether the running artifact was rebuilt and restarted after the fix.
This is especially important for AI systems that browse, ingest documents, call arbitrary URLs, or run generated code behind proxies. An application SBOM can be correct while a base runtime or sidecar still contains the vulnerable decoder. Close the deadline with an exercised data path and an artifact digest, not a ticket that says the package was upgraded.
Stay Sharp: one score can contain several measurement systems
Generalizability theory treats a score as the result of several facets, such as the task, rater, occasion, instrument, or site. It estimates how much variance belongs to each facet and how dependable the combined score is for the decision you want to make. The National Academies’ assessment guide explains the method as a way to test how raters, item types, and occasions affect score dependability. (National Academies)
Apply that lens to an agent benchmark. Let the observed score for model m on task t, laboratory l, and run r be:
score = model + task + lab + run + interactions + residual
Repeated attempts estimate run variance. Tasks across instruments and domains expose task variance. Recreating a subset in another lab reveals whether the result travels across procedures and labels. Interactions matter most: a model may look strong overall but fail consistently on one instrument or when one control type is missing.
The design question is not only whether the average is high. It is whether the decision remains stable across the facets your deployment will vary. If most uncertainty comes from repeated runs, add replication or consensus. If it comes from task families, publish per-slice gates. If it comes from laboratory context, require local validation. More benchmark tasks do not fix a missing facet.
Today’s three signals point to the same operating model. Adoption needs workflow outcomes. Scientific assistance needs evidence-linked gates. Vulnerability closure needs the identity of the decoder that runs. Treat the aggregate score as an index into the underlying system, not as permission to stop investigating it.