In this article
A three-month randomized trial with 133 practicing patent lawyers produced two different answers to the question, “Did AI help?” It improved the patent drafts lawyers delivered with assistance. It did not produce the same average improvement in junior lawyers’ judgment when the tool was removed.
The trial covered eleven US intellectual property firms. Two-thirds of the lawyers received a custom Google drafting assistant, while the rest received AI training but no tool access during the study. Blinded patent professionals scored the work. Assisted drafting quality rose by 0.34 standard deviations after 10 days and 0.38 after 90 days. (NBER)
At 90 days, every participant also redlined a flawed patent without AI. The treated group scored 0.32 standard deviations above the control group, but the result came entirely from senior lawyers, who improved by 0.45. Junior lawyers had no discernible average gain. Their scores spread toward both better and worse outcomes. (Google Research)
This is evidence from one profession, one custom tool, and three months. The study was funded by Google, several authors work for Google, and participating firms had regular non-exclusive business with the company. It does not establish that AI erodes junior expertise in every domain. It does show why an output metric cannot stand in for a learning metric.
Adoption needs two scorecards
Most AI adoption programs measure the artifact produced while the tool is available: quality, completion time, cost, acceptance, or error rate. Those measures answer whether the current system is useful. They do not answer whether people are becoming better at the underlying work.
Treat those as separate outcomes.
The delivery scorecard measures assisted work under the actual production configuration. It should include task quality, time, rework, severe errors, and reviewer effort. The capability scorecard measures what people can diagnose, decide, or repair without assistance, or with a deliberately limited form of it. It should be sliced by experience level and task family because an average can hide opposite effects.
The patent trial demonstrates the difference. Junior lawyers received larger immediate drafting gains, yet did not show the senior group’s average improvement on the unassisted redline. The finding is not a reason to deny juniors access to useful tools. It is a reason to stop inferring skill development from higher assisted output.
For a software team, the transfer task might be an unaided incident diagnosis, design review, or dependency-risk assessment. For a support team, it might be recognizing when a policy exception requires escalation. For an analyst, it might be reconstructing the business definition behind a generated query. The task must exercise the judgment the organization expects people to retain.
Design the learning loop before assistance becomes invisible
The Google account offers a hypothesis about the senior result. Senior lawyers described using AI as a logic auditor. On the unassisted redlining task, they spent more time rebuilding claims, removing language that narrowed legal scope, and tying edits to legal doctrine. Junior lawyers tended to proceed linearly, spend time on lower-stakes prose, and sometimes diagnose a problem without executing the repair. These observations help explain the result, but they are not a separate randomized test of the mechanism. (Google Research)
A team can still use the pattern to design a falsifiable learning loop:
- Ask the practitioner to state the intended decision, risks, or repair before revealing the assistant’s answer.
- Preserve the disagreement between the initial judgment and the AI suggestion.
- Require the practitioner to explain which evidence changed the decision.
- Sample periodic tasks with AI removed or constrained.
- Review transfer results by tenure, task type, and error severity rather than only the overall mean.
This changes the product requirement. An assistant should not only make the immediate artifact easier to produce. For work where future judgment matters, it should expose enough of the decision structure for a practitioner to practice diagnosis, compare alternatives, and receive targeted feedback.
There is a trade-off. Tool-off tests consume scarce expert time and can feel artificial. Running them on every task would defeat the productivity gain. A risk-based sample is more defensible: test frequently for high-consequence judgments and developing practitioners, less frequently for routine transformations whose underlying skill is not strategically important.
Human review is a staffing model, not a checkbox
A broader production study helps explain why this issue is operational. Measuring Agents in Production reports 20 case studies and a survey of 86 practitioners responsible for deployed systems across 26 domains. Sixty-eight percent of the systems executed at most ten steps before human intervention, 70% relied on prompting existing models rather than weight tuning, and 74% depended primarily on human evaluation. (PMLR)
Those figures describe current practice, not a causal recipe for success. They also expose a capacity dependency. If bounded agents rely on people for evaluation and intervention, the reviewers’ judgment is part of the production system. Review queues, turnover, training, and expertise loss can become reliability constraints even when model quality improves.
Capacity planning should therefore include both review demand and reviewer development. Record which task classes require expert intervention, how long decisions wait, which errors reviewers catch, and how performance changes when reviewers work without the assistant’s recommendation. A falling intervention rate can mean the system improved. It can also mean reviewers are over-trusting it or no longer see the cases that teach the boundary.
Model migrations can invalidate a longitudinal result
Long-running adoption studies also need a stable treatment. Google’s October 8 Gemini API changelog says requests to the deprecated gemini-3.7-flash are now automatically routed to gemini-3.8-flash, while gemini-3.5-flash requests route to gemini-3.6-flash. The older Deep Research preview is scheduled to shut down on October 23. (Gemini API)
A workplace experiment that measures behavior at day 10 and day 90 cannot assume the same assistant was present at both checkpoints. Pin an exact model when the provider supports it. Otherwise record the resolved model, system prompt, retrieval sources, tool policy, and relevant interface changes for every assessment. If the treatment changes, analyze the periods separately instead of attributing the difference to worker learning.
This is not only a research concern. Organizations use pilot results to choose staffing, training, and control designs. A moving model alias can turn a learning conclusion into a mixture of human adaptation and provider updates.
Stay Sharp: assistance is not an independent second opinion
A 2023 study of professional radiologists, revised in August 2026, shows another reason that “human plus AI” is not automatically safer. The researchers varied whether radiologists received AI predictions and contextual patient information. AI predictions did not improve diagnostic performance on average, while contextual information did. The authors found that radiologists underweighted AI predictions and failed to account fully for correlation between their own signal and the model’s signal. (Agarwal et al.)
If a person and model make similar errors from the same image, dataset, or protocol, combining their votes does not create two independent checks. If the reviewer misunderstands the model’s calibration, displaying a probability can add confidence without adding accuracy.
The design decision is therefore case routing before interface decoration. Estimate where the human has contextual information the model lacks, where the model has a validated advantage, and where their errors are correlated. Some cases should go to the person, some to the model, and some to a combined workflow with explicit disagreement handling. “Human in the loop” names a topology. It does not establish the quality of the loop.
What to watch
- Replications of the patent result in other professions, with independent tools, longer follow-up, and transfer tasks defined before deployment.
- Whether junior outcomes converge, remain split, or deteriorate after six and twelve months of routine assistance.
- Whether providers expose resolved model versions reliably enough for organizations to audit longitudinal workforce experiments.