Moonshot AI Introduces Seer to Accelerate Synchronous Reinforcement Learning Rollouts

Advancements in Efficient AI Training Pipelines

In the evolving landscape of artificial intelligence, reinforcement learning (RL) for large language models (LLMs) faces significant computational hurdles, particularly in handling long-sequence reasoning tasks. As models grow in scale and complexity, the rollout phase—where agents generate trajectories for policy updates—often becomes the primary bottleneck, consuming up to 87% of iteration time due to memory-intensive key-value (KV) cache management and variable output lengths. Researchers from Moonshot AI and Tsinghua University have developed Seer, an online context learning system designed to optimize this phase in synchronous, on-policy RL setups without altering the underlying algorithms. By restructuring request scheduling and leveraging shared context across responses, Seer addresses inefficiencies in GPU utilization, potentially enabling faster iteration cycles for training advanced reasoning models.

Challenges in Synchronous RL for Reasoning Models

Synchronous RL, commonly used in on-policy methods like Group Relative Policy Optimization (GRPO), requires all rollouts to complete before proceeding to the training step, ensuring data freshness but amplifying delays from stragglers. In experiments with models such as Moonlight (65,536-token context), Qwen2 VL 72B (40,960 tokens), and Kimi K2 (98,304 tokens), workloads were distributed across 32 compute nodes equipped with 8 H800 GPUs each. Tasks varied in scale, utilizing 32 to 256 GPUs, with 400 to 800 prompts per iteration generating 8 to 16 responses each.

The core issue stems from long chain-of-thought outputs, where KV cache sizes balloon from hundreds of megabytes to tens of gigabytes per request. This leads to reduced concurrency on individual instances, frequent preemptions, and re-decoding overhead. Notably, the tail latency—the time for the slowest 10% of requests—accounts for up to 50% of total rollout duration in baseline systems like veRL, which uses vLLM for inference. Such imbalances result in underutilized GPUs and prolonged training timelines, limiting scalability for production-grade RL on large models.

• Memory Fragmentation: Progressive token generation exacerbates KV cache growth, forcing systems to throttle batch sizes or evict states, which incurs recomputation costs.
• Load Imbalance: Fixed group assignments (sets of responses sharing a prompt) cause stragglers, as output lengths vary widely within groups.
• Iteration Dominance: Rollouts comprise 63% to 87% of overall time, making even modest tail reductions critical for throughput.

Core Mechanisms of Seer and Their Implementation

Seer builds on established infrastructure, including Mooncake’s disaggregated KV cache architecture—a two-tier DRAM and SSD store shared across nodes—and vLLM for inference, paired with Megatron for distributed training. It maintains identical RL algorithms to veRL, using only current-iteration data for on-policy fidelity. The system introduces three interconnected mechanisms orchestrated via a request buffer, context manager, and inference engine pool connected to a global KV cache pool.

Divided Rollout decomposes GRPO groups into individual requests, then further into fixed-size chunks (e.g., 8,000 tokens). At chunk boundaries, requests are re-enqueued and migrated across instances without re-prefilling, as KV states persist in the global pool. This fine-grained approach sustains high memory utilization (avoiding preemptions) while balancing loads, yielding up to 35% throughput gains in isolation.

Context-Aware Scheduling exploits correlations in output lengths among responses to the same prompt. For each group, one speculative request is prioritized in a shortest-first queue to gauge potential tail behavior. The context manager updates group-level length estimates based on completed tokens (or conservatively uses max limits), then applies a longest-first policy for remaining requests. This online adaptation approximates an oracle scheduler—aware of all lengths in advance—reducing tail exposure without prior knowledge. Adaptive Grouped Speculative Decoding enhances long-tail efficiency via a Distributed Grouped Draft Server (DGDS), which builds compressed suffix trees aggregating token patterns across a group.

Instances fetch these trees periodically for local speculation, adjusting draft depths and multi-path branching based on batch size, model type (dense or Mixture-of-Experts), and acceptance rates. In low-concurrency tail phases, deeper drafts boost accepted tokens per step. Ablation studies show this component adds 30% to 40% further speedup, with the full system achieving 77% to 87% overall iteration improvements.

• Global KV Cache Benefits: Enables seamless migrations, cutting re-decoding by preserving prefills across 32+ nodes.
• Speculative Efficiency: Grouped drafting leverages shared patterns, increasing tokens/second for correlated sequences.
• Preserved Guarantees: No off-policy data leakage, ensuring reproducible RL outcomes.

Performance Metrics and Broader Implications

Evaluated over 10 rollout iterations on the three models, Seer delivered 74% to 97% higher throughput (output tokens per second) compared to the veRL baseline, alongside 75% to 93% reductions in tail latency. For memory-constrained scenarios, the baseline’s tail-dominated time (up to 50%) was largely eliminated, allowing consistent GPU occupancy. These gains stem from exploiting GRPO’s structural similarities—prompt-shared responses exhibit predictable length and pattern correlations—without hardware upgrades. In the context of AI infrastructure trends, Seer’s focus on systems-level optimizations aligns with growing demands for efficient RL in reasoning tasks, where compute costs can exceed millions per run.

By mitigating KV cache fragmentation and scheduling inefficiencies, it could lower barriers for scaling on-policy methods, potentially accelerating advancements in agentic AI and multi-step decision-making. However, adoption may hinge on integration with existing stacks; while verifiable on H800 clusters, performance on newer architectures like H100s remains untested in these experiments (flagged as an uncertainty for extrapolation). As RL integrates deeper into LLM pipelines, systems like Seer highlight the shift toward holistic optimization, blending algorithmic stability with runtime efficiency. Would you integrate such context-aware scheduling into your AI training workflows to tackle similar bottlenecks?