Reportman.AI Logo Reportman.AI

← All articles

How can a big AI model run as fast as a small one?

· Toni Martir, arquitecto de software

An accounting firm setting up AI on its own computer finds models with names like Gemma 4 26B A4B. The 26B is its size: about 26 billion parameters (the numbers the model learned in training). You would expect it to be slower than a 4-billion-parameter model, yet Google, which built it, says it runs "almost as fast" as one.

The key is the A, for active. It is a mixture-of-experts model: according to its model card, it is split into 128 blocks, called experts, and for each token (a piece of a word) it uses 8 plus a shared one: 3.8 of its 25.2 billion parameters. Models that always use every parameter are called dense. But answering has two phases, reading what you send and writing the reply, and the savings differ.

A mixture-of-experts model against two dense ones: where it wins and where it doesn't

Writing, against one of its size Qwen3-32B, on an NVIDIA A10 card
4.5 times faster
Reading, against one of its size The same Qwen3-32B, with a 512-token text
Twice as fast
Writing, against a small one Qwen2.5-Coder-7B: the same on a Mac M2 Ultra, twice as fast on an NVIDIA DGX Spark
As fast or faster
Reading a short text, against a small one 2,048 tokens: 1.6 times faster on the Mac and 1.3 on the DGX Spark
Faster
Reading with a lot of text already read, against a small one 16% slower with 16,384 tokens already read on the Mac, and 8% with 32,768 on the DGX Spark
Slower

Tokens per second of Qwen3-30B-A3B, a mixture of experts, against each dense model on the same machine. Against the one of its size, measurements by a user in the llama.cpp repository, with both models compressed to 4 bits; against the small one, measurements by the author of llama.cpp with the coding versions of both, compressed to 8 bits.

When writing, it keeps the pace of what it uses

To write each token, the chip fetches the parameters from memory, and the pace is set by that transfer, not by the arithmetic. A dense model fetches all of them for every token; a mixture-of-experts model, only the active ones. That is why it writes far faster than a dense model of its size and, in measurements by llama.cpp's author, as fast as a small one or faster.

When reading, the advantage shrinks

Before writing anything, the model reads everything you send: the prefill, which depends on computing power, as I explained in why local AI is slow to answer. A mixture-of-experts model also computes only with its active experts, and reads faster than a dense model of its size, though it gains less than when writing.

Also, the experts replace only part of each layer: attention, which relates each token to all the previous ones, is used in full and weighs more the more text it has read. With a lot of text, the mixture-of-experts model reads slower than the small dense one. And asking a database means a lot of reading: the description of the tables goes with every question, and giving it only the one for the question's area shortens it.

But it takes memory like a big one

The price is memory, as Google warns next to its memory table: the 26B A4B must have all its parameters loaded, which is why it needs nearly as much as a dense model of its size. With each parameter stored in 4 bits instead of 16, it takes 14.4 GB, against 17.5 for the dense 31-billion model and 4.5 for the small E4B. And it is nearly as accurate as the dense one, without matching it: in Google's tests the 31-billion model wins all 15 that both took, as in MMLU Pro, reasoning questions across many subjects: 85.2% to 82.6%.

So how can it run that fast?

When writing, because it fetches only the active experts from memory: it writes as fast as a small dense model or faster. When reading, it also computes with them alone, but attention is used in full: with short texts it wins, and with long ones it falls behind the small model. And it needs nearly the memory of a dense model of its size.


At Reportman AI, the graphics card and processor tables on the local AI page show which models fit in each amount of memory, including some mixtures of experts: Gemma 4 26B A4B, from 16 GB on the card or 24 GB of memory without one, and Qwen3.6 35B A3B, from 24 GB on the card or 32 GB without one. With the Agent running as a service, any machine with AI capability can be shared by granting access by email: wherever it is, while it is connected it shows up in the pool of machines of everyone with access, who picks which one gets each question. On every plan, with no subscription fee: you provide the machines.

Sources

Toni Martir, arquitecto de software · · sources checked on