Memory
25 updates on Memory.
MobiLLM fine-tunes OPT-1.3B in 4.5 GB on device by moving backpropagation to a server
Researchers fine-tuned OPT-1.3B on a Jetson Xavier NX in 4.5 GB by keeping the frozen model on the device and the trainable side network on a server.
Meta's ParetoQ compares five bit widths and puts the accuracy cliff at 1 bit
Meta trained the same models at 1, 1.58, 2, 3 and 4 bits under one recipe and reports 1.58-bit to 3-bit beating 4-bit at matched memory.
Spectra ships ternary 3.9B models that match 4-bit quantisation at half the bits
Nolano AI trained 54 models from 99M to 3.9B parameters on the same 300B tokens in ternary, quantised and half-precision form to compare them by bit size.
Hugging Face releases SmolLM at 135M, 360M and 1.7B parameters
Three base models trained on the newly released SmolLM-Corpus, with published memory footprints from 109.78 MB to 3422.76 MB.
TensorOpera releases Fox-1, a 1.6B model trained on 3 trillion tokens
TensorOpera published a 1.6B model under Apache 2.0 and reports it ahead of Gemma-2B and Qwen1.5-1.8B on a six-benchmark average.
PowerInfer-2 runs a 47B model on a OnePlus 12 at 11.68 tokens per second
Shanghai Jiao Tong University researchers report a 47B model decoding at 11.68 tokens per second on a OnePlus 12, with weights streamed from flash.
One shared on-device LLM keeps a context per app and switches in 0.27 seconds
Peking University, BUPT and Tsinghua researchers compress and swap each app conversation context in 16-token chunks, switching up to 20 times faster.
MeRino designs sub-100M language models that run 4.9 times faster than OPT-350M
Researchers design 52M to 64M parameter transformers by maximising entropy under a compute budget, matching OPT-350M accuracy on an NVIDIA Jetson Nano.
BitNet b1.58 gives every weight three values and runs 3B in 2.22 GB
Microsoft trained models whose every weight is -1, 0 or 1, which replaces multiplication with addition, and reports parity with full-precision Llama from 3B.
Meta's MobileLLM trades width for depth and gains 2.7 and 4.3 points
Meta Reality Labs built 125M and 350M models around deep and thin layers and profiled them on an iPhone 13 through ExecuTorch.
Apple researchers run models twice the size of available DRAM from flash
The LLM in a flash paper loads parameters from flash on demand and reports 4 to 5 times faster CPU and 20 to 25 times faster GPU inference.
RWKV trains like a transformer and runs with constant memory per token
The paper releases pretrained RNN weights from 169M to 14B parameters and reports constant time and memory per token during inference.