SERE: Similarity-based Expert Re-routing for Efficient MoE Batch Decoding

This skill enables Claude to help users deploy and configure SERE, a method that accelerates Mixture-of-Experts (MoE) model inference by dynamically reducing active experts during batch decoding. SERE pre-computes similarity matrices between experts via calibration, then at inference time re-routes tokens from secondary experts to their most similar primary experts — skipping redundant computation without permanent pruning or retraining. It integrates into vLLM as a plugin with a custom CUDA kernel, achieving up to 2.0x decoding speedup with minimal quality loss.

When to Use

• When a user wants to reduce latency or increase throughput of MoE model serving (Qwen1.5-MoE, DeepSeek-V2-Lite, Qwen3-30B-A3B)
• When deploying MoE models via vLLM and batch decoding is the bottleneck (memory-bound decoding stage)
• When the user asks about expert pruning or merging alternatives that don't require retraining
• When optimizing a production MoE inference pipeline for cost efficiency under latency SLAs
• When a user wants to trade a small accuracy margin for significant decoding speed gains on reasoning benchmarks
• When configuring select_top_k and threshold hyperparameters for a SERE deployment

Key Technique

The core problem: In MoE models, each token activates only a few experts (sparse activation). But during batched inference, different tokens in a batch may route to different experts, causing the union of activated experts to grow large — negating the sparsity benefit and making the decoding stage memory-bound.

SERE's solution operates in three stages per MoE layer: