AI on your own machine: how much memory does the graphics card need?
· Toni Martir, arquitecto de software
A company decides to run AI on a machine of its own instead of buying it from a provider (the reasons are in how much is your data privacy worth?). Two almost identical quotes come in, both with an RTX 5060 Ti graphics card. But NVIDIA makes that card with either 8 or 16 GB of memory, and that half line in the quote decides which AI can run on each computer.
That memory belongs to the card, separate from the computer's own, and spec sheets list it as VRAM. The card speeds up whatever part of the model fits in it, so it has to hold the whole model, what it reads to answer you and whatever the operating system already uses.
What the model takes up
A model is a huge list of learned numbers, called parameters: a «12B» model has about 12 billion of them. They can be stored compressed, at lower precision: at 4 bits instead of 16, they take a quarter of the space. Google's 12-billion Gemma 4 model thus drops from 26.7 GB to 6.7. Compression comes at a price: at lower precision a model is less capable, though it may be enough.
What it reads while it answers
Google warns those figures leave out the context, everything the model reads in one go: your question, the conversation and its own answer. When the AI queries your database, the context also carries the description of the tables, which in a large database is a lot of text, as I explained when writing about RAG. And the context takes up room on the card too: as the Qwen3 documentation measures, their 14-billion model, compressed, went from about 10 GB reading a single token (the word fragments models count in) to more than 15 reading 30,720.
What the operating system already uses
On Windows, the desktop is drawn by the graphics card: as Microsoft explains, every open window gets video memory so the system can compose the screen. That share is no longer free for the AI. Some laptops have two graphics chips: the one built into the processor, which drives the screen, and the dedicated card, which has no display connected, so its memory stays free for the model. And on a Linux server with no graphical desktop, nothing draws a screen with the card: its memory is left for the AI.
Which models fit, roughly, in each amount of memory
From the Reportman AI local AI compatibility table. Each row also takes the models in the rows above it. With 32 or 48 GB the same models fit as with 24, with more room left for the context; with 96, GPT-OSS 120B fits as well.
If it does not fit
If the model does not fit entirely, it still runs. Programs such as llama.cpp, which runs these models on your own machine, split the work: the card speeds up the part that fits in its memory and the processor does the rest, without that speed-up. With no card, the processor does it all.
So how much memory does it need?
As much as your chosen model needs, compressed, plus its context, plus whatever the desktop uses if the card also drives the screen. So you choose the model first and the card second: with the RTX 5060 Ti from the start, the 16 GB version fits models the 8 GB one cannot. With less memory the model still runs, but the card only speeds up part of it.
In Reportman AI, local AI can be used on every plan, on your own computer or on one machine with a graphics card for the whole office: you install the Agent on it as a service and grant access by email to whoever will use it. The Agents form a distributed network, so that card can be used from anywhere, the web app included. The local AI page has the full table, for NVIDIA, AMD or Intel cards and for the computer's memory when there is no card, with slower answers. No subscription fee: you provide the machine. And if buying one does not pay off, there are plans with cloud models.
Sources
- NVIDIA, GeForce RTX 5060 Family, RTX 5060 Ti specs: «Standard Memory Config: 16 GB / 8 GB GDDR7».
- Google DeepMind, Gemma 4 (the official Gemma documentation, which the old ai.google.dev page now points to), table «Gemma 4 Inference Memory Requirements»: Gemma 4 12B, 26.7 GB in BF16 (16-bit) and 6.7 GB in Q4_0 (4-bit). On precision: «Models with lower parameters and bit counts (lower precision) have less capabilities, but may be sufficient for your AI task.» And on the context: «They don't include the additional VRAM needed for supporting software or the context window.»; «Larger context windows require significantly more VRAM on top of the base model weights.»
- Qwen, Speed Benchmark (the documentation the official Qwen3 repository points to for memory requirements): «We test the speed and memory usage when generating 2048 tokens, with input lengths of 1, 6144, 14336, 30720, 63488, and 129024 tokens.» Table «Qwen3-14B (Transformers)», AWQ-INT4 quantization: 9962 MB with 1 input token and 15323 MB with 30720.
- Microsoft, DirectX Developer Blog, GPUs in the task manager (Steve Pronovost, lead engineer for the Windows GPU memory manager, 2017): «On discrete GPUs this is your VRAM, the memory that sits on your graphics card.»; and «when an application creates a top level window, video memory will be allocated to hold the content of that window. That video memory surface is […] shared with the desktop window manager (dwm.exe) such that the window can be composed onto the desktop.»
- Microsoft Learn, Using cross-adapter resources in a hybrid system: «The integrated GPU is integrated into the CPU chipset and outputs to an integrated display panel such as an LCD panel.»; «The discrete GPU is a render-only device, and no display outputs are connected to it.»
- llama.cpp, official repository, «Description» section: «CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity»; and «1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use».
- Reportman AI: Local AI (the compatibility table by card and computer memory, local AI on every plan, the Agent as a service with access granted by email, the distributed network, the web setup with local AI through the Agent, and acceleration on NVIDIA, AMD and Intel) and plans.
Toni Martir, arquitecto de software · · sources checked on