Hugging Face adds native GGUF inference to Transformers, starting with Qwen3.5 on Macs
Hugging Face added native support in Transformers for GGUF, the quantized file format used by llama.cpp, so developers can load those checkpoints through the library’s standard from_pretrained() API instead of switching to a separate runtime. Engineers Marc Sun, Arthur Zucker and Lysandre wrote in the announcement that on Apple Silicon, weights stay packed in their GGUF blocks and Metal GPU kernels run the matmuls directly on the compressed data, skipping a full dequantization step. The feature currently covers Qwen3.5, including its dense and MoE variants, plus compatible Qwen3.8 checkpoints.
The packed inference path draws its Metal kernels from a kernels library that fetches them from the Hub, including ggml-attn, the same flash attention implementation llama.cpp runs for both prefill and decode. Separate kernels handle quantization, normalization and the gated delta net used in the hybrid attention layers of both Qwen3.5 and Qwen3.8. Architectures outside this initial set, or devices other than Apple Silicon, fall back to a legacy loader that dequantizes GGUF weights into a dense model at load time, a path that already covers Llama, Mistral, Qwen2, Qwen2Moe, Phi3, Bloom, Falcon, StableLM, GPT2 and Starcoder2, among others.
On a MacBook Pro M2 Max with 32 GB of memory, Hugging Face reports about 70 tok/s generating with Qwen3.5-4B at Q4_K_M quantization, close to the 72 tok/s the company measured for llama.cpp on the same hardware. The company says its new layer kernels add 51 to 109 percent more throughput over the packed-quantization-only path, depending on the model, with Qwen3.5-4B gaining 59 percent, Qwen3.8-27B 51 percent and Qwen3.5-35B-A3B 109 percent. Further gains come from generation-loop changes such as dropping unneeded attention masks early and checking stop conditions asynchronously. Quantization options for the 4B model range from a 2.74 GB Q4_K_M file up to an 8.42 GB BF16 file.
The packed path requires Transformers’ main development branch together with a matching version of the kernels library, and Hugging Face says it is limited to MPS devices for now, with padding and batched requests still needing optimization. Models can also be served through an OpenAI-compatible API with transformers serve, which identifies a GGUF checkpoint as <repo>:<file>.gguf since a single Hub repository can hold several quantizations of the same model.