How many people can an AI serve at once?
· Toni Martir, arquitecto de software
An accountancy firm puts AI on a computer of its own, with a 16 GB graphics card, for the whole office. At nine, five people ask it a question at the same time. One gets an answer; the other four wait in line. The machine isn't broken: it is set up to serve one person at a time.
How many people it serves at once comes down, above all, to the card's memory, because the AI keeps a separate space there for each person, holding everything it reads to answer them. Whoever doesn't fit waits their turn, and those served at the same time get their answers more slowly.
Five questions at nine, one 16 GB card
An office example. How much context each person needs depends on their question and on how long the description of the tables they query is.
Out of the box, one person at a time
In Ollama, one of the programs that run AI models on a machine of your own, the setting that decides how many requests each model handles at once (its slots, so to speak) is set to one, according to its documentation. The rest wait in a queue.
The number of slots can be raised, but every extra slot costs memory and speed.
Each person, their own context
The context is everything the model reads in one go to answer: the question, the conversation and, if it is querying your database, the description of the tables, which in a large database is a lot of text, as I explained when discussing RAG. Each person served at the same time needs their own. Ollama gives an example: a context of about 2,000 tokens (the word fragments models count in) with four requests at once becomes one of about 8,000.
In Qwen's measurements, one of its models, compressed, took up more than 5 GB extra when reading 30,720 tokens instead of one. With four people at once, each with that context, Ollama's rule multiplies that memory by four: more than 20 GB, more than the firm's entire card. If the memory won't stretch that far, the context that fits is split between the slots, and each person gets less.
Speed gets shared too
According to NVIDIA's documentation, as the number of simultaneous requests grows, the machine produces more text in total, but each person gets less per second.
And the same documentation says when the queue forms: when requests exceed the most the program can process at once, multiplied by the number of copies of the model that are running. More slots on the same card share out what is there; another copy of the model, on another card or another machine, adds capacity.
So how many people does it serve at once?
As many as it has slots configured. And it has room for as many slots as there are full contexts that fit in the memory left over by the model and the desktop, if the card also draws the screen. Out of the box, one; the rest wait their turn, and those served together go more slowly. If the line grows at nine and the memory has room for more slots, you raise them; if it doesn't, the answer is another machine.
At Reportman AI any machine with AI capacity can be shared: you install the Agent on it as a service and grant access by email to whoever is going to use it, from the desktop app or the web. Machines running the Agent form a distributed network: wherever they are, in the office, at another branch or at home, as long as they are connected they show up as available in the pool of machines of each person with access, who picks which one gets each question. When one can't keep up, you add another to the pool and people spread out between the two. According to the local AI page, it works on every plan: no subscription, you supply the machines. If buying another one doesn't pay off, there are plans with cloud models.
Sources
- Ollama, FAQ, section «How does Ollama handle concurrent requests?»: «Parallel request processing for a given model results in increasing the context size by the number of parallel requests. For example, a 2K context with 4 parallel requests will result in an 8K context and additional memory allocation.»; «OLLAMA_NUM_PARALLEL - The maximum number of parallel requests each model will process at the same time, default 1. Required RAM will scale by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH.»
- Qwen, Speed Benchmark (the documentation the official Qwen3 repository points to for memory requirements), table «Qwen3-14B (Transformers)», AWQ-INT4 quantization: 9962 MB with 1 input token and 15323 MB with 30720.
- NVIDIA, NIM Benchmarking, Metrics: «As the number of concurrent requests increases, total system TPS increases while TPS per user decreases as latency increases.» (TPS is tokens per second.)
- NVIDIA, NIM Benchmarking, Parameters and Best Practices: «If the concurrency exceeds the maximum batch size multiplied by the number of active replicas, some requests wait in a queue for later processing.»
- Reportman AI: local AI (local AI on every plan, the distributed network of machines with AI capability, the Agent as a service with access granted by email, and local AI through the Agent in both the desktop and the web setup) and plans.
Toni Martir, arquitecto de software · · sources checked on