UQO Working Papers · WP33 · v0.1
TITAN-MoE: Topology-Aware and Predictive Expert Orchestration for Distributed Mixture-of-Experts Inference on Apple Silicon
A Measured Systems Study on a Heterogeneous Mac Cluster
- Year
- 2026
- First release
- November 2026
- Latest revision
- November 2026
- Publication status
- Working paper
Bounded expert residency is free; prefetch, fused kernels and gigabit remote tiers are not
TITAN-MoE matches mlx-lm bit for bit (489 vs 494 tok/s), yet the best causal predictor only ties demand loading, kernel fusion moves nothing at batch one, and remote experts over 1 GbE cost 2.5 ms per layer; only the expert-contiguous storage layout wins (8 to 36 tok/s).
Abstract
Sparse mixture-of-experts (MoE) language models touch only a few experts per token, so a model whose weights exceed a machine's memory can still be served if the right experts are in the right place at the right time. This preliminary working paper reports the design, implementation and measurement of TITAN-MoE, an MLX-native inference runtime for Apple Silicon that manages expert residency with bounded per-layer slot tables, loads experts on demand from local NVMe, prefetches them with causal predictors and, in its distributed mode, executes them on other Macs of a heterogeneous cluster connected by gigabit Ethernet. Every design choice is validated against mlx-lm on identical weights: with all experts resident the runtime is bit-identical in logits and within one percent of mlx-lm's decode throughput, and with bounded residency the router's choices and the logits are unchanged by construction.
Five hypotheses (causal prefetch, topology-aware placement, hierarchical caching, fused Metal kernels, and adaptive orchestration on the latency–throughput–energy frontier) are tested with a pre-registered protocol on a live inventory of a 21-node Mac cluster. The results of version 0.1 are mostly negative and fully attributed: on the small model, the best causal predictor (a per-layer logistic probe, recall@8 of 0.76, within 0.16 of the oracle hit rate) only ties with demand loading because installing a prefetched expert costs as much graph time as the read it saves; kernel fusion is immaterial because decode at batch size one is launch-bound at 28% of memory bandwidth; and executing half the experts on a second Mac Studio over gigabit Ethernet costs 2.5 ms per layer, more than a cold-NVMe miss, so the remote tier is a capacity mechanism rather than a latency one.
The storage layout, by contrast, matters: an expert-contiguous record file raises cold-NVMe decoding from 8 to 36 tokens per second. Energy measurements and the adaptive scheduler are pending and labelled as such; every reported number has a raw log and the next gates (deeper prefetch horizons, the medium model on nodes that cannot hold it, cost-aware placement) are fixed by the protocol.
Key findings
489 vs 494 tok/s
TITAN all-resident vs mlx-lm, bit-identical logits
0.76
recall@8 of the best causal predictor, 0.16 below oracle
8 → 36 tok/s
cold-NVMe decode with expert-contiguous storage layout
2.5 ms
per-layer cost of remote experts over 1 GbE
28%
of memory bandwidth used by launch-bound batch-one decode
21
Macs in the live cluster inventory, all on gigabit Ethernet
Scientific contributions
- An exact-semantics MoE runtime for MLX
- Expert registry over safetensors byte ranges, pread NVMe tier, per-layer slot tables with LRU/LFU eviction, an exact execution block over gather_qmm and an asynchronous prefetcher with four causal predictors; placement decides speed, never which experts run.
- Correctness and zero-overhead baseline
- Bit-identical logits (max |Δ| = 0.0) and token-identical greedy decoding versus mlx-lm on OLMoE-1B-7B-0125 4-bit at 64/32/16/8 slots; all-resident decode at 489 tok/s against 494 for mlx-lm, and 121.6 vs 121.7 tok/s on Qwen3-30B-A3B.
- Causal prefetch does not pay on the small model
- One-layer-ahead prefetch raises the hit rate from 0.48 to 0.63 but lowers throughput; the paper derives the lead-time condition d·τ_layer ≥ λ_tier and shows the best predictor (recall@8 0.76) only ties demand loading (35.0 vs 36.4 tok/s).
- Kernel fusion is immaterial at batch one
- Decode is launch-bound at about 28% of nominal memory bandwidth, so fusing the up and gate projections changes throughput by ±2%.
- The gigabit remote tier adds capacity, not speed
- Executing odd-numbered experts on a second Mac Studio over 1 GbE keeps logits bit-identical but drops OLMoE decoding from 70–87 to 15 tok/s (2.5 ms per layer, 20× a cold-NVMe miss); a LAN fetch of one expert costs about 28 ms, ruling out moving weights during decode.
- Storage layout is the one clear win
- An expert-contiguous record file raises cold-NVMe decoding at 16 slots from 8 to 36 tok/s, more than any prediction or caching policy tested.
Methodology
Methodology (5)
Pre-registered protocol with gates A–G
Five hypotheses stated before measurement, a frozen protocol at Gate C, and a research log; nothing is reported without a raw log, and pending gates (energy, adaptive scheduler) are labelled as such.
Exact comparison against mlx-lm
Same weights, quantization and kernels; numerical tests check logits and greedy tokens, and llama.cpp and colibri are reported as references rather than like-for-like competitors.
Trace-driven simulator and analytic cost model
A simulator calibrated on measured constants (τ_layer = 0.128 ms, NVMe 8.5 GB/s random reads, 0.3 ms expert install) and a formal placement–prefetch problem with a deterministic cost-aware heuristic.
Statistical audit with bootstrap intervals
Medians and 95% bootstrap confidence intervals (10,000 resamples) for every throughput comparison and its paired difference.
Two-node TCP transport
An authenticated transport with a bit-exact remote expert worker measured between two Mac Studios (M3U96a and M3U96b) on the cluster's gigabit Ethernet.
Research data
- Models and workloads
- OLMoE-1B-7B-0125-Instruct and Qwen3-30B-A3B-Instruct-2507 as pinned mlx-community 4-bit conversions; synthetic prompt workloads written for the study and the routing traces derived from them (no external datasets).
- Cluster inventory and benchmark logs
- A sanitized live inventory of 21 reachable Macs (M1–M6, 8–96 GB; iperf3 0.945 Gbit/s, 0.5–1.0 ms RTT, no Thunderbolt link) and immutable raw decode-throughput logs under benchmarks/raw with SHA-256 manifests, 12–18 runs per configuration.
Reproducibility
- Runtime in titan/ with pytest correctness tests against mlx-lm that write evidence to benchmarks/raw/tests/
- Benchmark scripts (bench_decode, collect_traces, simulate_cache, ssd_bandwidth) with immutable raw logs and artifacts/manifest.json listing every artifact's SHA-256
- Figures and tables regenerated from benchmarks/raw by scripts/make_figures.py and scripts/make_tables.py; pinned requirements-lock.txt and pinned model revisions
- Full LaTeX source of the 30-page paper; prior-art matrix of 59 verified works in research/prior_art.csv; AI-assistance disclosure in the reproducibility appendix
Research artifacts
Version history
| Version | Date | Description | File |
|---|---|---|---|
| v0.1current | November 2026 | Current manuscript as published on www.spboucher.ai (revised edition after the 2026-10-17 scientific review for papers 1–17). | PDF · 30 p. |