WP33
TITAN-MoE: Topology-Aware and Predictive Expert Orchestration for Distributed Mixture-of-Experts Inference on Apple Silicon
A Measured Systems Study on a Heterogeneous Mac Cluster
Sparse mixture-of-experts (MoE) language models touch only a few experts per token, so a model whose weights exceed a machine's memory can still be served if the right experts are in the right place at the right time. This preliminary working paper reports the design, implementation and measurement of TITAN-MoE, an…
Artificial intelligencemixture of expertsinference systemsApple Siliconexpert offloadingPreliminary draft PDF · 30 pages