In this article
NVIDIA and Princeton’s Dao AI Lab changed what a hybrid model stores between tokens. Instead of updating and writing an entire state-space model state after every token, ReplaySSM keeps a checkpoint plus a bounded buffer of recent inputs. The checkpoint is refreshed only when the buffer fills.
That bookkeeping change produced a 1.20x to 1.47x end-to-end decoding speedup over tuned vLLM across four hybrid models at batch size 256. The isolated state-space kernels improved by 1.46x to 1.80x. The same report says the method provided no speedup at batch size 1, where other costs dominate.12
This is not evidence that every hybrid model just became 47 percent faster. It is evidence that recurrent state traffic can become the bottleneck at serving-scale concurrency, and that a state representation should be designed for the operations the serving engine actually performs.
Fixed-size state can still carry a large bandwidth bill
A state-space model, or SSM, compresses prior tokens into a fixed-size recurrent state. That avoids a key-value cache that grows with context length, but fixed size does not mean cheap access.
In conventional decoding, each layer loads its state from high-bandwidth memory, applies the current token’s update, uses the result to produce an output, and stores the new state. NVIDIA reports that one FP32 state for a Mamba-2 layer in the evaluated 120B-class Nemotron model is 4 MiB per sequence. The model also has five times as many SSM layers as attention layers. Under the paper’s batch-256 B300 setup, the SSM path remained the largest part of a decode step through 100,000 tokens of context.1
The limiting resource is memory traffic. The update performs relatively little arithmetic for every byte moved, so adding compute units does not remove the constraint. Over a replay window of L tokens, conventional decoding reads the state L times and writes it L times. ReplaySSM still reads the checkpoint L times, because every output needs it, but writes the full state once. Recent token inputs are much smaller than the state matrix, so the dominant traffic is roughly halved.2
The honest summary is therefore narrower than “cache inputs, not state.” ReplaySSM keeps both. It makes the full state durable less often.
One representation improves rollback and precision isolation
The checkpoint-and-buffer representation changes three serving paths.
- Standard decoding: a non-flush step appends the current SSM inputs and computes the output from the checkpoint plus the buffer. A flush folds the buffer into a new checkpoint. The evaluated window was 16 tokens, chosen as the best setting across the four tested models.1
- Speculative decoding: draft-token inputs enter the buffer before verification. Accepting a prefix advances a pointer; rejected records are overwritten. The engine no longer needs a full state snapshot for each draft position. At batch size 512 on two 120B-class models, the reported throughput was 1.85x and 2.10x that of vLLM’s speculative path, with matched acceptance lengths.1
- State quantization: most outputs can read a low-precision checkpoint while recurrent updates start from a separate high-precision checkpoint. Low-precision read error can affect current outputs, but it is not fed back into later state updates. On GPQA-Diamond, three of four evaluated models stayed within about two percentage points of their FP32-state accuracy under the reported separate-precision settings; the fourth, the 4B Nemotron model, still lost more than five points.1
These benefits share a cause, but they are not interchangeable. A team can gain from fewer writes without adopting low-precision state. Speculative throughput also depends on acceptance length, draft width, concurrency, and available GPU memory. Treat each path as a separate release decision.
Benchmark the operating point, not the method name
ReplaySSM is already integrated into TensorRT-LLM, vLLM, SGLang, and FlashInfer, according to the authors. NVIDIA also says GLM-5.3-Flash uses the method in production and that Kimi-K3 uses the related input-caching approach for draft verification.1 Those integrations make testing practical. They do not settle whether the change improves a particular service.
An adoption test should preserve the workload dimensions that determine the result:
- Concurrency: report batch-size buckets, not only an aggregate. The published advantage grew with batch size and disappeared at batch size 1.
- Context and generation length: separate prefill, time to first token, inter-token latency, and completed-request throughput. A decode optimization should not borrow credit from a different phase.
- Model structure: record the SSM-to-attention layer ratio and state size. The result applies to evaluated Mamba-2 and Gated DeltaNet hybrids, not automatically to every recurrent architecture.
- Speculation: hold the drafter, verification width, and acceptance-length distribution constant. Faster rollback cannot compensate for a poor drafter.
- Quality: compare output divergence and task accuracy by generation-length bucket. A global average can hide error that accumulates only on long reasoning traces.
Use the current serving stack as the baseline, including its best kernel configuration. Otherwise the experiment measures integration quality as much as the representation.
Application watch: automatic repair is still a release system
Kore.ai published its Autoloop technical account and announced general availability on October 7. An October 12 report repeated the launch. The vendor describes a loop that evaluates an agent against seven configured goals, traces a failure to a routing rule, contract, fixture, prompt, or policy construct, proposes a targeted change, and re-runs the broader goal set before keeping or rolling back the change.34
The useful distinction is not that an AI system can optimize another AI system. It is that the product exposes three change-authority modes: Autopilot applies verified changes, Copilot requires approval, and Advisor only recommends. Kore.ai also states the core limitation plainly: weak tests create weak evidence, and missing tools, data, or policy decisions remain human work.3
For production teams, a passing regression gate should therefore be necessary but not sufficient for autonomous application. Route changes involving identity, permissions, money, regulated decisions, or irreversible actions need a stricter authority boundary than copy or retrieval-tuning changes. The repair engine’s confidence and the organization’s permission to deploy are different variables.
Stay Sharp: use the roofline before adding compute
The roofline model asks how much work a kernel performs for each byte it moves. NVIDIA’s profiling guide calls this ratio arithmetic intensity. Low-intensity kernels sit under the memory-bandwidth slope; high-intensity kernels can approach the compute ceiling.5
Use that model before choosing an optimization:
- Measure bytes moved and operations performed for the target kernel.
- Locate the kernel relative to the memory and compute limits of the actual GPU.
- If it is memory-bound, reduce traffic, improve locality, or increase useful work per load before buying more FLOPS.
- Re-profile at production batch sizes. The same kernel can move between regimes as concurrency changes.
ReplaySSM is a clean example. It does not accelerate the bottleneck by performing the same state writes faster. It changes the state representation so most writes no longer happen.
Watch next
- Independent results across more hybrid architectures, GPU generations, and low-concurrency workloads.
- Tail-latency and per-request results alongside aggregate throughput.
- Accuracy by generation length for separate state precisions outside the three reported benchmarks.
- Whether automatic agent-repair products publish change-type-specific rollback and escape rates, not only aggregate goal scores.