UltraEP: Real-Time Load Balancing for Large-Scale MoE Training and Inference
Core Highlights
Xiaohongshu and Peking University jointly proposed UltraEP, the first to bring "real-time load balancing based on exact routing information" into production systems. It dynamically replicates hot experts per microbatch and per layer, and on large models such as Qwen3-235B, training throughput averages 94.6% of ideal performance—a 42% gain over Megatron-LM—while inference prefill throughput improves 1.56x over SGLang. This provides an efficient engineering solution for large-scale MoE training and inference, attacking one of the most stubborn sources of wasted compute in modern model serving.
The significance is economic as much as technical: in large clusters, even a few percent of idle compute compounds into millions of dollars of wasted silicon per year. By pushing utilization close to the theoretical ceiling, UltraEP turns a hidden tax into reclaimed capacity that the same hardware budget can now spend on more users or larger models.
What Happened
The core of a Mixture-of-Experts (MoE) model is "routing"—each token is assigned to a few experts for computation. The problem is that different tokens favor experts extremely unevenly, causing some experts to be overloaded while others sit idle and GPU compute is wasted. Traditional schemes either statically partition experts or rely on approximate statistics for load balancing, with limited precision. UltraEP's breakthrough is "exactness": using true routing probabilities, it knows in advance how many tokens will hit each expert, then dynamically replicates hot experts onto idle devices per microbatch and per layer to flatten the load.
This matters because MoE was sold partly on the promise of cheap scaling—only a fraction of parameters activate per token—but that promise is silently broken when the active experts pile onto a few devices. UltraEP restores the original economic logic of the architecture by enforcing balance at the granularity where imbalance actually occurs, namely every layer of every microbatch.
Technical Details
Dynamic replication is the key to UltraEP. Because MoE routing differs at every layer and every microbatch, fixed expert placement cannot cope with fluctuation; before each forward pass, UltraEP computes each expert's load from exact routing information, replicates high-load experts to idle cards for parallel processing, and merges low-load ones. This "scheduling by real traffic" avoids approximation error and pushes training throughput close to the theoretical ceiling.
On super-large MoEs like Qwen3-235B, 94.6% of ideal performance means the waste from load skew is almost eliminated, recovering capacity that previously required buying more hardware to obtain and letting the same cluster serve more users at the same latency. The method's reliance on exact routing rather than estimates is what separates a durable fix from a heuristic that works only on certain workloads.
Comparison with Competitors
Against Megatron-LM and SGLang, UltraEP's advantage comes from "exact routing plus fine-grained dynamic replication." Megatron-LM's expert parallelism is relatively static and efficiency drops noticeably under routing skew; SGLang is fast on the inference side but its prefill stage is still dragged by expert imbalance. UltraEP incorporates the real routing of production environments into scheduling, thus delivering tangible gains of 42% and 1.56x on large models rather than relying on micro-optimizations that barely move the needle.
The contrast shows that system-level insight into the model's own behavior can beat generic parallelism tricks that treat every expert as equally busy when they plainly are not. Where others guessed, UltraEP measures; where others pre-allocated, it adapts. That philosophical difference is why the gains hold on real, messy traffic instead of clean benchmarks.
Industry Impact and Use Cases
Simply put, MoE is the mainstream architecture for cutting costs and boosting efficiency in today's large models, but "expert imbalance" has always been a hidden cost of deployment. UltraEP lets the same hardware deliver higher throughput, directly amortizing training and inference costs, and is especially critical for businesses requiring frequent fine-tuning and long-context inference.
Born from Xiaohongshu's own high-concurrency recommendation and generation scenarios, it carries industrial-grade validation rather than only bench results. If open-sourced or standardized, it will lower the barrier for the whole industry to deploy super-large MoEs, letting more teams afford billion- and trillion-parameter models without proportionally larger clusters and without silently paying for idle silicon that the architecture was supposed to save them from using in the first place.
If adopted widely, the technique could quietly reshape cloud economics for the MoE era. Training and serving bills are among the largest line items for AI companies, and a method that reclaims double-digit percentages of wasted compute is effectively a discount on every future model, paid for by better software rather than by procuring yet more hardware. Efficiency, increasingly, is the feature that decides who can afford to keep training at the frontier. Whoever trains cheapest can train most often, and frequency compounds into better models. In the MoE era, therefore, software efficiency is becoming the new scaling law for the entire industry.