In this article
A fixed decision model completed 42.9% of the tested ALFWorld tasks when it chose actions itself. Researchers then froze the model, showed it complete trajectories, and asked which actions had contributed to the goal. Those scores helped train a Qwen2.5-1.5B policy that reached 82.9% success, compared with 72.8% for the outcome-only baseline. (Liu et al.)
The model did not become a better actor. Its information changed. At action time it could see only the task history so far and was often drawn toward an easy subgoal before completing a prerequisite. After the episode, later observations exposed which earlier steps had actually enabled success. The result is a useful warning for agent architecture: acting, judging, and teaching are different jobs even when one model can technically perform all three.
The evidence is promising but bounded. The paper is a preprint, studies ALFWorld and WebShop, and reports policy results across three evaluation seeds. It does not establish that the method transfers to production agents, open-ended software work, or safety-critical control. It does show exactly where a coarse reward loses information and how an offline reviewer can recover some of it.
Ask about contribution, not only completion
The paper compared two questions for scoring actions. A value question asked whether the task would eventually succeed. A progress question asked whether the target action helped achieve the goal. Under hindsight context, the value question scored milestone actions only 0.05 higher than no-ops. The progress question widened that gap to 0.36.
That distinction changed training. For the 1.5B policy on ALFWorld, hindsight plus value reached 69.3% success. Hindsight plus progress reached 82.9%. On WebShop, the same progress configuration reached 71.4%, versus 56.8% for the outcome-only baseline. The teacher added its centered action scores to the GRPO outcome advantage during training, then disappeared at inference time. The authors report teacher API costs between $22.21 and $30.07 for the four training configurations they ran, using the price available during the experiment. (Paper) (Code)
For a production team, the immediate lesson is not “put a judge behind every tool call.” Hindsight is unavailable when the actor must choose the next action. Using future context in an online gate would leak information the runtime cannot possess. Instead, separate the paths:
- The actor executes a versioned task contract. Express prerequisites as ordered state transitions when order matters. A generic reminder to consider dependencies did not fix the actors in the study.
- The recorder preserves the trajectory. Keep tool requests, observations, state deltas, policy versions, and terminal outcomes. A final answer alone cannot support step-level review.
- The judge runs after the episode. Ask whether each action contributed to progress, not merely whether the final run succeeded.
- The teacher updates offline behavior. Use held-out tasks to decide whether a new policy, prompt, router, or skill version deserves promotion. Keep runtime guardrails independent of the learning signal.
Teams that do not fine-tune can still use this split. An offline judge can rank traces for review, identify repeated no-ops, and turn useful milestones into regression cases. The deployment gate should still depend on task outcomes, side effects, and human-defined policy checks rather than the judge’s score alone.
Application radar: Amplitude closed the loop from observation to experiment
Amplitude’s Wave product agent starts from product telemetry rather than a backlog item. Teams give it a product area, goals, and metrics. Wave examines analytics, session replays, customer feedback, and code, then produces an opportunity, a proposed solution, a technical plan, and acceptance criteria that can be sent to a coding agent. The same system follows the change into an experiment and reads the result. (OpenAI customer story)
On Amplitude’s homepage, Wave reviewed a sample of 100 session replays. Eighty people typed their own question while 12 used a generic suggestion card. Wave proposed personalized cards based on recent charts, dashboards, or the referring page, while keeping free-form input and a generic fallback. After human review, a coding agent implemented the change. Amplitude reports that card-driven messages rose 160% relative to control after one day.
That is evidence of a working discovery-to-measurement loop, not proof of durable business impact. The readout covers one day, the page does not disclose the experiment population, and the account is a joint customer story from OpenAI and Amplitude. The 160% figure measures a channel-specific engagement action, not retention, revenue, or user success.
The more transferable evidence is in the handoff design. Wave passes a technical plan and acceptance criteria to the coding agent, and teams choose automation by change class. Amplitude says most bug fixes can go directly to coding agents, while new features and larger bets keep human review. It also evaluates models per task rather than declaring one universal winner. In its Data Science Agent evaluation, GPT-6.1 Sol scored 67.5 against 69.3 for another leading model, but its mean model cost was $1.28 per dispatch instead of $2.63.
The architectural pattern is a controlled product loop:
observation -> hypothesis -> plan -> implementation -> experiment -> measured outcome
Each arrow needs a durable artifact. Without the replay sample, acceptance criteria, code revision, experiment assignment, and metric definition, a system can optimize for activity while losing the evidence that connects activity to value.
Operations radar: Natera made skills the unit of blame
Natera’s SWARM system processes more than 65 daily lab-operations tickets across three Jira queues. AWS and Natera report that average first response fell from 17.4 hours to about 8 minutes over a 90-day production evaluation, while 96.2% of incoming tickets received an automated diagnostic report before engineer review. (AWS case study)
The important design choice is smaller than the headline. Natera maintains 33 versioned skills: 24 instrument-specific skills and nine cross-cutting workflow skills. Deterministic Python handles tasks such as parsing Jira fields and formatting reports. Language-model skills interpret symptoms, match documentation, and write diagnostic narratives. Each skill has an interface and can be tested independently. When a diagnosis fails, the team can identify a skill and a knowledge source instead of debugging one large prompt.
Natera also separated two runtime profiles. A background agent optimizes for throughput and attaches a structured report to a ticket. An interactive agent supports engineers who need images, follow-up questions, and alternative paths. Both use the same skill library, but their latency and context requirements do not force one execution policy.
The quality numbers require care. Of 524 tickets in the evaluation set, 269 had both a confidence label and an engineer-verified resolution. High-confidence cases reached 74% accuracy across 154 tickets. Moderate and low confidence both reached 65%, across 78 and 37 tickets respectively. Natera says the low-confidence sample is too small for a strong conclusion. Adding missing vendor manuals for the weakest instrument family moved accuracy from 71% to 75% without changing the model.
So 96.2% is coverage, not correctness. The production lesson is to publish both. Measure how often the system responds, how quickly it responds, how often engineers can verify the answer, and how accuracy changes by skill, equipment family, and confidence tier. Fast, broad coverage is useful only when the error surface remains visible.
Stay Sharp: credit assignment changes what the system can learn
An outcome reward answers whether a trajectory worked. Credit assignment asks which decisions caused that outcome. When every action in a successful run receives the same signal, a necessary prerequisite, an irrelevant detour, and a lucky final step can all look equally good.
Hindsight credit assignment uses later events to estimate the contribution of an earlier action. The idea predates language agents and appears in reinforcement-learning research that conditions credit on what happened after a decision. (Harutyunyan et al.) The current paper applies that logic to tool-using trajectories with a frozen decision model.
A practical review should preserve three boundaries:
- No future leakage at runtime. The actor can use only information available at the decision point.
- Explicit milestones after the run. The reviewer needs state changes that distinguish progress from plausible narration.
- Independent release evidence. A hindsight score can guide learning, but held-out task success, side effects, and policy compliance decide promotion.
The larger point is role design. Choose an actor for bounded execution, a judge for the information it can extract from completed traces, and a teacher for the behavior it improves offline. Benchmark rank alone does not tell you which role a model should hold.