Reportman.AI Logo Reportman.AI

← All articles

How many people can an AI serve at once?

· Toni Martir, arquitecto de software

An accountancy firm puts AI on a computer of its own, with a 16 GB graphics card, for the whole office. At nine, five people ask it a question at the same time. One gets an answer; the other four wait in line. The machine isn't broken: it is set up to serve one person at a time.

How many people it serves at once comes down, above all, to the card's memory, because the AI keeps a separate space there for each person, holding everything it reads to answer them. Whoever doesn't fit waits their turn, and those served at the same time get their answers more slowly.

Five questions at nine, one 16 GB card

One slot, as it comes out of the box It answers one person. The other four wait in line, each until the ones ahead have finished
They wait
Five slots, each with its full context All five are answered at once, more slowly, and the memory for what they read is multiplied by five
Five times the memory
Five slots in the same memory All five are answered at once, more slowly, and each gets a fifth of the context: it may not be enough for the description of the tables
Less context
Two machines Each person asks one of the two, and each machine serves its own line
More capacity

An office example. How much context each person needs depends on their question and on how long the description of the tables they query is.

Out of the box, one person at a time

In Ollama, one of the programs that run AI models on a machine of your own, the setting that decides how many requests each model handles at once (its slots, so to speak) is set to one, according to its documentation. The rest wait in a queue.

The number of slots can be raised, but every extra slot costs memory and speed.

Each person, their own context

The context is everything the model reads in one go to answer: the question, the conversation and, if it is querying your database, the description of the tables, which in a large database is a lot of text, as I explained when discussing RAG. Each person served at the same time needs their own. Ollama gives an example: a context of about 2,000 tokens (the word fragments models count in) with four requests at once becomes one of about 8,000.

In Qwen's measurements, one of its models, compressed, took up more than 5 GB extra when reading 30,720 tokens instead of one. With four people at once, each with that context, Ollama's rule multiplies that memory by four: more than 20 GB, more than the firm's entire card. If the memory won't stretch that far, the context that fits is split between the slots, and each person gets less.

Speed gets shared too

According to NVIDIA's documentation, as the number of simultaneous requests grows, the machine produces more text in total, but each person gets less per second.

And the same documentation says when the queue forms: when requests exceed the most the program can process at once, multiplied by the number of copies of the model that are running. More slots on the same card share out what is there; another copy of the model, on another card or another machine, adds capacity.

So how many people does it serve at once?

As many as it has slots configured. And it has room for as many slots as there are full contexts that fit in the memory left over by the model and the desktop, if the card also draws the screen. Out of the box, one; the rest wait their turn, and those served together go more slowly. If the line grows at nine and the memory has room for more slots, you raise them; if it doesn't, the answer is another machine.


At Reportman AI any machine with AI capacity can be shared: you install the Agent on it as a service and grant access by email to whoever is going to use it, from the desktop app or the web. Machines running the Agent form a distributed network: wherever they are, in the office, at another branch or at home, as long as they are connected they show up as available in the pool of machines of each person with access, who picks which one gets each question. When one can't keep up, you add another to the pool and people spread out between the two. According to the local AI page, it works on every plan: no subscription, you supply the machines. If buying another one doesn't pay off, there are plans with cloud models.

Sources

Toni Martir, arquitecto de software · · sources checked on