How can a big AI model run as fast as a small one?
· Toni Martir, arquitecto de software
An accounting firm setting up AI on its own computer finds models with names like Gemma 4 26B A4B. The 26B is its size: about 26 billion parameters (the numbers the model learned in training). You would expect it to be slower than a 4-billion-parameter model, yet Google, which built it, says it runs "almost as fast" as one.
The key is the A, for active. It is a mixture-of-experts model: according to its model card, it is split into 128 blocks, called experts, and for each token (a piece of a word) it uses 8 plus a shared one: 3.8 of its 25.2 billion parameters. Models that always use every parameter are called dense. But answering has two phases, reading what you send and writing the reply, and the savings differ.
A mixture-of-experts model against two dense ones: where it wins and where it doesn't
Tokens per second of Qwen3-30B-A3B, a mixture of experts, against each dense model on the same machine. Against the one of its size, measurements by a user in the llama.cpp repository, with both models compressed to 4 bits; against the small one, measurements by the author of llama.cpp with the coding versions of both, compressed to 8 bits.
When writing, it keeps the pace of what it uses
To write each token, the chip fetches the parameters from memory, and the pace is set by that transfer, not by the arithmetic. A dense model fetches all of them for every token; a mixture-of-experts model, only the active ones. That is why it writes far faster than a dense model of its size and, in measurements by llama.cpp's author, as fast as a small one or faster.
When reading, the advantage shrinks
Before writing anything, the model reads everything you send: the prefill, which depends on computing power, as I explained in why local AI is slow to answer. A mixture-of-experts model also computes only with its active experts, and reads faster than a dense model of its size, though it gains less than when writing.
Also, the experts replace only part of each layer: attention, which relates each token to all the previous ones, is used in full and weighs more the more text it has read. With a lot of text, the mixture-of-experts model reads slower than the small dense one. And asking a database means a lot of reading: the description of the tables goes with every question, and giving it only the one for the question's area shortens it.
But it takes memory like a big one
The price is memory, as Google warns next to its memory table: the 26B A4B must have all its parameters loaded, which is why it needs nearly as much as a dense model of its size. With each parameter stored in 4 bits instead of 16, it takes 14.4 GB, against 17.5 for the dense 31-billion model and 4.5 for the small E4B. And it is nearly as accurate as the dense one, without matching it: in Google's tests the 31-billion model wins all 15 that both took, as in MMLU Pro, reasoning questions across many subjects: 85.2% to 82.6%.
So how can it run that fast?
When writing, because it fetches only the active experts from memory: it writes as fast as a small dense model or faster. When reading, it also computes with them alone, but attention is used in full: with short texts it wins, and with long ones it falls behind the small model. And it needs nearly the memory of a dense model of its size.
At Reportman AI, the graphics card and processor tables on the local AI page show which models fit in each amount of memory, including some mixtures of experts: Gemma 4 26B A4B, from 16 GB on the card or 24 GB of memory without one, and Qwen3.6 35B A3B, from 24 GB on the card or 32 GB without one. With the Agent running as a service, any machine with AI capability can be shared by granting access by email: wherever it is, while it is connected it shows up in the pool of machines of everyone with access, who picks which one gets each question. On every plan, with no subscription fee: you provide the machines.
Sources
- Google DeepMind, Gemma 4 model card, "Mixture-of-Experts (MoE) Model": for the 26B A4B, "Total Parameters 25.2B", "Active Parameters 3.8B" and "Expert Count 8 active / 128 total and 1 shared"; and "The "A" in 26B A4B stands for "active parameters" in contrast to the total number of parameters the model contains. By only activating a 4B subset of parameters during inference, the Mixture-of-Experts model runs much faster than its 26B total might suggest. This makes it an excellent choice for fast inference compared to the dense 31B model since it runs almost as fast as a 4B-parameter model."
- Google DeepMind, Gemma 4 model card, "Benchmark Results": MMLU Pro, 85.2% for the 31B and 82.6% for the 26B A4B. The table gives results for both models in 15 tests, and the 31B scores better in all 15 (in OmniDocBench, where "lower is better", 0.131 against 0.149).
- Google DeepMind, Gemma 4 model overview, "Gemma 4 Inference Memory Requirements", column "Q4_0 (4-bit)": E4B, 4.5 GB; 26B A4B, 14.4 GB; 31B, 17.5 GB. And under "Key Considerations for Memory Planning": "While it only activates 4 billion parameters per token during generation, all 26 billion parameters must be loaded into memory to maintain fast routing and inference speeds. This is why its baseline memory requirement is much closer to a dense 26B model than a 4B model."
- NVIDIA Technical Blog, Mastering LLM Techniques: Inference Optimization: on generation, "The speed at which the data (weights, keys, values, activations) is transferred to the GPU from memory dominates the latency, not how fast the computation actually happens. In other words, this is a memory-bound operation."; on prefill, "Each new token depends on all the previous tokens".
- NVIDIA Technical Blog, Applying Mixture of Experts in LLM Architectures: "In many published papers, the MoE technique is applied to Multi-Layer Perceptrons (MLPs) within transformer blocks. In this case, the MLP within the transformer block is usually replaced with a set of expert MLP subnetworks".
- Qwen, Qwen3.6-35B-A3B: "Number of Parameters: 35B in total and 3B activated", "Hidden Layout: 10 × (3 × (Gated DeltaNet → MoE) → 1 × (Gated Attention → MoE))" and "Number of Experts: 256".
- llama.cpp, issue 13652 (20 May 2025),
measurements by a user with
llama-benchon an "NVIDIA A10-24Q": Qwen3-32B Q4_0 (17.41 GiB),pp512672.53 andtg12823.22; Qwen3-30B-A3B Q4_0 (16.42 GiB),pp5121328.67 andtg128103.50 tokens per second. A project member replies: "the 30B model is a Mixture-of-Experts […] which is why it's so much faster". - llama.cpp, measurements by the project's author (updated 5 February 2026),
Qwen3-Coder-30B-A3B Q8_0 against Qwen2.5-Coder-7B Q8_0, with
llama-bench: NVIDIA DGX Spark ("NVIDIA GB10"):tg3261.06 against 29.43;pp20482986.97 against 2250.28;pp2048 @ d327681348.17 against 1471.23. Mac M2 Ultra:tg3278.97 against 79.68;pp20482453.11 against 1565.91;pp2048 @ d16384741.04 against 886.46.ppis reading (prefill),tgis writing anddis the tokens already read. - Yubo Wang et al., MMLU-Pro (arXiv): "a More Robust and Challenging Multi-Task Language Understanding Benchmark", with "more challenging, reasoning-focused questions".
- The arithmetic: writing, 103.50 / 23.22 = 4.5; 61.06 / 29.43 = 2.1; 78.97 / 79.68 = 0.99. Reading, 1328.67 / 672.53 = 2.0; 2986.97 / 2250.28 = 1.3; 2453.11 / 1565.91 = 1.6; with text already read, 1348.17 / 1471.23 = 0.92 and 741.04 / 886.46 = 0.84.
- Reportman AI: local AI (the graphics card and processor tables with the models for each amount of memory, local AI on every plan, the distributed network and the Agent as a service with access by email) and plans.
Toni Martir, arquitecto de software · · sources checked on