Redesign Mixture-of-Experts Routers with Manifold Power Iteration

arXiv:2606.12397v1 Announce Type: cross Abstract: Router is the cornerstone component to the Mixture-of-Experts models. Serving as expert proxies, the rows of the router matrix compute their similarity to the MoE inputs to determine which subset of experts is activated. Ideally, each router row is designed to encode the expert matrix into this...

Read the full article at the source.