Reportman.AI Logo Reportman.AI

← All articles

Why is AI on your own machine slow to answer?

· Toni Martir, arquitecto de software

At an accounting firm with its AI on an office computer, someone asks «which customers are more than 90 days overdue?»: a while with nothing on screen, then the words come out at whatever pace the machine allows.

There are two phases with two limits. Prefill reads the whole context (the question, the conversation and the description of the tables) before the first word, and it is computation. Generation writes the answer token by token (word fragments), and what holds it back is bandwidth: the gigabytes per second the memory can deliver to the chip.

The same model on three machines: how long it takes to read and how fast it writes

Graphics card: RTX 5060 Ti 4,608 compute cores and 448 GB/s of memory
reads 3,737 t/s 4,000 tokens: 1 s writes 91 t/s
Unified memory: Apple M4 Max The chip's 40-core graphics reads and writes, with 546 GB/s
reads 886 t/s 4,000 tokens: 4.5 s writes 83 t/s
Desktop PC: Ryzen 9 3950X The 16-core processor alone, with dual-channel DDR4
reads 76 t/s 4,000 tokens: 53 s writes 9.6 t/s

Llama 2 7B compressed to 4 bits (3.56 GiB), from the measurements in the llama.cpp repository, in tokens per second (t/s): reading 512 tokens and writing 128. The time for 4,000 tokens is arithmetic at the measured speed; with longer texts, the speed drops.

Prefill is computation

NVIDIA describes prefill as a highly parallel matrix-by-matrix operation that fills the graphics card: the compute-bound phase. Whoever multiplies more at once wins, and a graphics chip does it with thousands of cores: 4,608 on an RTX 5060 Ti.

That is why a PC does not read like a Mac: on the M4 Max it is not the processor that reads, but the chip's graphics. AMD's Ryzen AI Max+ 395, also with unified memory, shows it: it reads at 300 tokens per second with 16 threads (tasks in parallel) on the processor, and at 1,487 with its integrated graphics. And a desktop Ryzen 9 3950X, on the processor alone, reads at 76: a 4,000-token table description takes it 53 seconds.

Generation is memory

Writing is matrix by vector, one token at a time: for each one, the whole model travels from memory to the chip, and the pace is set by that transfer, not by the arithmetic. In a llama.cpp example on a processor, generation stays at 16 tokens per second with 8 threads and does not rise with 32: the memory has nothing more to give.

A processor is not slow just because it is a processor: it depends on how it connects to memory. In the llama.cpp measurements, the M4 Max writes almost as fast as the RTX 5060 Ti, although it reads four times more slowly; the Ryzen 9 3950X, with dual-channel DDR4, writes at 9.6 tokens per second.

Why PC memory falls short

On a desktop PC, memory reaches the processor through few channels: two on the Core i9-14900K, which with DDR5 at 5,600 million transfers per second moves 89.6 GB/s. A graphics card has its own GDDR7 memory on a bus (the path to the chip) 128 bits wide: 448 GB/s on the RTX 5060 Ti; at 512 bits, the RTX 5090 reaches 1,792. Unified memory sits in the same package as the chip, and processor and graphics share it without copying data: 546 GB/s on the M4 Max. At 3.8 GB per token, 89.6 GB/s gives a ceiling of 23 tokens per second; 448 GB/s, one of 117.

So why is it slow to answer?

Because prefill, which depends on compute, decides how long the first word takes, and generation, which depends on bandwidth, the pace of the rest. A graphics card does well at both; a chip with unified memory writes almost as fast but reads more slowly; and a desktop PC without a card reads with the processor, far more slowly, and writes with a ceiling five times lower than the card's.


In Reportman AI, local AI uses the graphics card (CUDA or Vulkan) or, without one, the processor, which is slower; the local AI page lists the compatible cards and processors. With the Agent running as a service, any machine with AI capability can be shared by granting access by email. Machines running the Agent form a distributed network: wherever they are, as long as they are connected, they show up in the pool of machines of everyone with access, who picks which one gets each question. It works on every plan, with no subscription fee: you provide the machines; and with no card at all, the paid plans run in the cloud.

Sources

Toni Martir, arquitecto de software · · sources checked on