Back to the ticker

Apple runs an 816B model across four Mac Studios at 36 tok/s with MLX

Apple Machine Learning Research has split Muse Spark Open across a cluster of four Mac Studios and measured more than 36 tokens per second of generation on the setup. The model carries 816B parameters in total but activates only 42B per token, a mixture-of-experts design where just a fraction of the experts run for any given input. Apple describes the result as “well above average human reading speed.”

A single machine cannot hold the weights. Apple puts the parameters alone at more than 1.6TB of RAM in the model’s native BF16 precision, which 4-bit NVFP4 quantisation brings down to about 493GB, still above what fits in the 512GB of unified memory on the largest Mac Studio once a key-value cache and runtime overhead are added. The two smaller models in the same benchmark run do fit, with Apple reporting up to 68GB for Qwen 3.8 at 27B parameters and up to 158GB for the 128B Mistral Medium 3.5 at a 65,536-token context.

The link between the machines is new. macOS 26.2 added a low-latency RDMA driver over Thunderbolt 5, and MLX v0.30.1 and later builds communication collectives on top of it under the name jaccl. Apple wired each pair of machines peer-to-peer with two Thunderbolt cables to double the link bandwidth, which the company says cut time to first token for Muse Spark Open on four machines from about 50.5 seconds with one cable to 40.6 seconds at a 65,536-token context.

MLX offers two ways to split a model, and the distributed documentation covers both. Pipeline parallelism cuts the model across depth, so each Mac runs a slice of the layers and hands its output to the next, which Apple calls extremely simple to implement but which leaves decode speed essentially unchanged. Tensor parallelism cuts each layer across width, producing many small communication operations per layer, and Apple states that a low-latency interconnect like Thunderbolt RDMA is critical for the approach to yield any speedup at all. On a single machine Apple separately measures the M5 Ultra at up to 4.6× the time-to-first-token speed of an M3 Ultra and 1.6× its decode throughput, at a peak power draw of 300W against 200W.

Decode speed in tokens per second

BenchmarkFour M5 Ultra, tensorTwo M5 Ultra, pipeline
Muse Spark Open (816B, NVFP4)36.7315.74
Mistral Medium 3.5 (128B, FP8)23.377.70
Qwen 3.8 (27B, BF16)50.8919.69

All figures according to Apple, measured over 128 generated tokens following the prompt. Muse Spark Open does not fit on one machine, so its baseline is two Mac Studios with pipeline parallelism.

The pieces are public. MLX LM ships the mlx.launch helper that starts a job across nodes from a host file, the mlx_lm.benchmark command Apple used for its figures, and an mlx_lm.server mode that exposes an OpenAI-compatible endpoint on the cluster. Apple shows that server pointed at a local coding agent through an OpenCode provider config on port 8080. Both the chunked prefill that overlaps work across machines and the fast CPU-GPU synchronisation used in the benchmarks are enabled by flags, and Apple labels the latter an experimental MLX feature.

  1. Edge0 releases MoE expert-offloading framework, demos 35B model on iPhone
  2. Apple puts the M6 in a new 14-inch MacBook Pro with 25 percent faster AI workloads
  3. Meta AI builds MobileMoE, 5.3B parameters with 0.9B active per token