Why is AI on your own machine slow to answer?
· Toni Martir, arquitecto de software
At an accounting firm with its AI on an office computer, someone asks «which customers are more than 90 days overdue?»: a while with nothing on screen, then the words come out at whatever pace the machine allows.
There are two phases with two limits. Prefill reads the whole context (the question, the conversation and the description of the tables) before the first word, and it is computation. Generation writes the answer token by token (word fragments), and what holds it back is bandwidth: the gigabytes per second the memory can deliver to the chip.
The same model on three machines: how long it takes to read and how fast it writes
Llama 2 7B compressed to 4 bits (3.56 GiB), from the measurements in the llama.cpp repository, in tokens per second (t/s): reading 512 tokens and writing 128. The time for 4,000 tokens is arithmetic at the measured speed; with longer texts, the speed drops.
Prefill is computation
NVIDIA describes prefill as a highly parallel matrix-by-matrix operation that fills the graphics card: the compute-bound phase. Whoever multiplies more at once wins, and a graphics chip does it with thousands of cores: 4,608 on an RTX 5060 Ti.
That is why a PC does not read like a Mac: on the M4 Max it is not the processor that reads, but the chip's graphics. AMD's Ryzen AI Max+ 395, also with unified memory, shows it: it reads at 300 tokens per second with 16 threads (tasks in parallel) on the processor, and at 1,487 with its integrated graphics. And a desktop Ryzen 9 3950X, on the processor alone, reads at 76: a 4,000-token table description takes it 53 seconds.
Generation is memory
Writing is matrix by vector, one token at a time: for each one, the whole model travels from memory to the chip, and the pace is set by that transfer, not by the arithmetic. In a llama.cpp example on a processor, generation stays at 16 tokens per second with 8 threads and does not rise with 32: the memory has nothing more to give.
A processor is not slow just because it is a processor: it depends on how it connects to memory. In the llama.cpp measurements, the M4 Max writes almost as fast as the RTX 5060 Ti, although it reads four times more slowly; the Ryzen 9 3950X, with dual-channel DDR4, writes at 9.6 tokens per second.
Why PC memory falls short
On a desktop PC, memory reaches the processor through few channels: two on the Core i9-14900K, which with DDR5 at 5,600 million transfers per second moves 89.6 GB/s. A graphics card has its own GDDR7 memory on a bus (the path to the chip) 128 bits wide: 448 GB/s on the RTX 5060 Ti; at 512 bits, the RTX 5090 reaches 1,792. Unified memory sits in the same package as the chip, and processor and graphics share it without copying data: 546 GB/s on the M4 Max. At 3.8 GB per token, 89.6 GB/s gives a ceiling of 23 tokens per second; 448 GB/s, one of 117.
So why is it slow to answer?
Because prefill, which depends on compute, decides how long the first word takes, and generation, which depends on bandwidth, the pace of the rest. A graphics card does well at both; a chip with unified memory writes almost as fast but reads more slowly; and a desktop PC without a card reads with the processor, far more slowly, and writes with a ceiling five times lower than the card's.
In Reportman AI, local AI uses the graphics card (CUDA or Vulkan) or, without one, the processor, which is slower; the local AI page lists the compatible cards and processors. With the Agent running as a service, any machine with AI capability can be shared by granting access by email. Machines running the Agent form a distributed network: wherever they are, as long as they are connected, they show up in the pool of machines of everyone with access, who picks which one gets each question. It works on every plan, with no subscription fee: you provide the machines; and with no card at all, the paid plans run in the cloud.
Sources
- NVIDIA Technical Blog, Mastering LLM Techniques: Inference Optimization: on prefill, «this is a matrix-matrix operation that’s highly parallelized. It effectively saturates GPU utilization»; on generation, «This is like a matrix-vector operation that underutilizes the GPU compute ability compared to the prefill phase. The speed at which the data (weights, keys, values, activations) is transferred to the GPU from memory dominates the latency, not how fast the computation actually happens. In other words, this is a memory-bound operation.»
- NVIDIA Dynamo, Disaggregated Serving: «the memory-bound decoding phase» and «the computation-bound prefill phase».
- llama.cpp, llama-bench,
example «Different numbers of threads», processor only («llama 7B mostly Q4_0»): generation
(
tg 16) at 16.71 tokens per second with 8 threads and 16.41 with 32. The documentation does not say which machine was used. - llama.cpp, Performance of llama.cpp on Apple Silicon M-series,
table maintained by the project's author: M4 Max with a 40-core GPU and 546 GB/s, Llama 2 7B
in Q4_0 (3.56 GiB),
-p 512 -n 128 -ngl 99(the whole model on the graphics, with Metal): prefill at 885.68 tokens per second and generation at 83.06. Its chart sets «PP vs GPU Cores and TG vs Bandwidth». - llama.cpp, Performance of llama.cpp on Nvidia CUDA,
a table of user-submitted measurements, same model and test, without flash attention: RTX 5060
Ti, «16 GB / GDDR7 / 128 bit»,
pp512at 3737.25 andtg128at 90.94. - llama.cpp, Performance of llama.cpp with Vulkan,
user-submitted measurements, same model and test, without flash attention:
Ryzen AI Max+ 395
(2 October 2026), processor alone with 16 threads,
pp512at 300.27 andtg128at 27.68, and with its Radeon 8060S graphics over Vulkan (which reports «uma: 1», memory shared with the processor), 1487.08 and 51.81; Ryzen 9 3950X (14 June 2026), «16-Core Processor», «DDR4 Dual Channel at 3600 MT/s», processor alone with 16 threads,pp512at 75.84 andtg128at 9.57. - NVIDIA, GeForce RTX 50 Series comparison: RTX 5060 Ti with 4608 CUDA cores, a 128-bit bus and 448 GB/s; RTX 5090 with 512 bits and 1792 GB/s, both with GDDR7.
- Intel, Core i9-14900K: «Up to DDR5 5600 MT/s», 2 memory channels and «Max Memory Bandwidth 89.6 GB/s».
- Apple, MacBook Pro (14-inch, M4 Pro or M4 Max, 2024): «M4 Max with 16-core CPU and 40-core GPU (546GB/s memory bandwidth)»; and Apple unleashes M1: «a unified memory architecture that brings together high-bandwidth, low-latency memory into a single pool within a custom package. This allows all of the technologies in the SoC to access the same data without copying it».
- The arithmetic: 4,000 tokens divided by 3737.25, 885.68 and 75.84 tokens per second gives 1.07, 4.5 and 52.7 seconds; and, for the generation ceilings, 3.56 GiB is 3.82 GB, 89.6 / 3.82 = 23.4 and 448 / 3.82 = 117.2 tokens per second.
- Reportman AI: local AI (CUDA and Vulkan, the processor without a card with longer response times, the card and processor tables, local AI on every plan, the distributed network and the Agent as a service with access by email) and plans.
Toni Martir, arquitecto de software · · sources checked on