Reportman.AI Logo Reportman.AI

← All articles

AI on your own machine: how much memory does the graphics card need?

· Toni Martir, arquitecto de software

A company decides to run AI on a machine of its own instead of buying it from a provider (the reasons are in how much is your data privacy worth?). Two almost identical quotes come in, both with an RTX 5060 Ti graphics card. But NVIDIA makes that card with either 8 or 16 GB of memory, and that half line in the quote decides which AI can run on each computer.

That memory belongs to the card, separate from the computer's own, and spec sheets list it as VRAM. The card speeds up whatever part of the model fits in it, so it has to hold the whole model, what it reads to answer you and whatever the operating system already uses.

What the model takes up

A model is a huge list of learned numbers, called parameters: a «12B» model has about 12 billion of them. They can be stored compressed, at lower precision: at 4 bits instead of 16, they take a quarter of the space. Google's 12-billion Gemma 4 model thus drops from 26.7 GB to 6.7. Compression comes at a price: at lower precision a model is less capable, though it may be enough.

What it reads while it answers

Google warns those figures leave out the context, everything the model reads in one go: your question, the conversation and its own answer. When the AI queries your database, the context also carries the description of the tables, which in a large database is a lot of text, as I explained when writing about RAG. And the context takes up room on the card too: as the Qwen3 documentation measures, their 14-billion model, compressed, went from about 10 GB reading a single token (the word fragments models count in) to more than 15 reading 30,720.

What the operating system already uses

On Windows, the desktop is drawn by the graphics card: as Microsoft explains, every open window gets video memory so the system can compose the screen. That share is no longer free for the AI. Some laptops have two graphics chips: the one built into the processor, which drives the screen, and the dedicated card, which has no display connected, so its memory stays free for the model. And on a Linux server with no graphical desktop, nothing draws a screen with the card: its memory is left for the AI.

Which models fit, roughly, in each amount of memory

6 GB Such as the 6 GB RTX 3050
Qwen3.5 4B Gemma 4 4B
8 GB Such as the RTX 5060
Qwen3.5 9B Ornith 1.5 9B
12 GB Such as the RTX 5070
Gemma 4 12B
16 GB Such as the RTX 5080 or the RX 9070 XT
Gemma 4 26B A4B GPT-OSS 20B
24 GB Such as the RTX 4090 or the RX 7900 XTX
Gemma 4 31B Qwen3.8 27B Qwen3.6 35B A3B and four more

From the Reportman AI local AI compatibility table. Each row also takes the models in the rows above it. With 32 or 48 GB the same models fit as with 24, with more room left for the context; with 96, GPT-OSS 120B fits as well.

If it does not fit

If the model does not fit entirely, it still runs. Programs such as llama.cpp, which runs these models on your own machine, split the work: the card speeds up the part that fits in its memory and the processor does the rest, without that speed-up. With no card, the processor does it all.

So how much memory does it need?

As much as your chosen model needs, compressed, plus its context, plus whatever the desktop uses if the card also drives the screen. So you choose the model first and the card second: with the RTX 5060 Ti from the start, the 16 GB version fits models the 8 GB one cannot. With less memory the model still runs, but the card only speeds up part of it.


In Reportman AI, local AI can be used on every plan, on your own computer or on one machine with a graphics card for the whole office: you install the Agent on it as a service and grant access by email to whoever will use it. The Agents form a distributed network, so that card can be used from anywhere, the web app included. The local AI page has the full table, for NVIDIA, AMD or Intel cards and for the computer's memory when there is no card, with slower answers. No subscription fee: you provide the machine. And if buying one does not pay off, there are plans with cloud models.

Sources

Toni Martir, arquitecto de software · · sources checked on