How Much VRAM Do I Need to Run a Local LLM? Check the Model and Context First

For local text chat, 8 GB of VRAM is a reasonable starting point; 12–16 GB gives larger models and longer conversations more room. A 24 GB card offers more options, but a 5.2 GB model download alone cannot tell you whether the session will fit on an 8 GB card.

The card’s capacity is not all available to the model. A long conversation uses additional memory, other applications may already occupy the GPU, and a model running partly from system RAM may respond differently from one that stays on the card. Choose an exact model build and test your usual workload before shopping for more memory.

How much VRAM do common local LLMs need?

Start with the model download and allow additional space for the live session. Try a compact quantized model first on 8 GB. With 12–16 GB, you can test a larger candidate; 24 GB leaves room to investigate bigger downloads. These are planning bands, not verified minimum VRAM requirements for every model or runtime.

A model described as “8B” has roughly eight billion parameters, but that figure does not tell you how those weights are stored. Quantization shrinks their footprint by using lower-precision representations. Hugging Face’s bitsandbytes documentation explains the principle, though its specific implementation is not a measurement of every Ollama build. Two files with the same parameter count can demand different amounts of storage and working memory.

The Ollama tags below list published download sizes and advertised context ceilings. They do not report measured peak VRAM use, and a listed context ceiling does not guarantee that your hardware can run a session at that length.

Those entries come from the Qwen3 tag page and the Llama 3.1 tag page. Ollama displays rounded GB sizes, while GPU utilities may use GiB. The figures can rule out obvious mismatches, but they cannot predict the last free megabyte on your card.

The 5.2 GB qwen3:8b file is worth testing on an 8 GB GPU, though the entire session may not stay on the card. The 9.3 GB qwen3:14b download exceeds that capacity before the conversation starts. A 20 GB download on a 24 GB card also leaves little room for the running session. The 43 GB llama3.1:70b download cannot keep that build’s weights wholly on one 24 GB card.

File size does not rank writing quality. For email-sized questions, a smaller model that responds promptly may suit you better than a larger one that takes much longer to load.

Why does a model need more memory than its download size?

The downloaded file holds the model weights. During a conversation, the runner also maintains a key-value, or KV, cache and uses working buffers; other GPU applications may take a share. A long document can increase memory use without changing the model file. Leave room for the running session.

The KV cache keeps information from tokens the model has processed so it can continue the conversation. Ollama’s context-length documentation puts the trade-off bluntly: “Setting a larger context length will increase the amount of memory required to run a model.” There is no universal fixed allowance to tack onto every model, because model architecture, cache format, batch settings and runner behavior vary.

Consider the same 8B build used for a brief recipe question and for a long contract followed by an hour of questions. Its weights stay the same, but the contract conversation needs more context. An advertised maximum such as 40K describes the tag’s available window, not the amount Ollama necessarily allocates on your machine. Its context page describes defaults that vary by VRAM, while another FAQ entry gives a different general default. Check the active context reported by your installation instead of assuming either default applies.

A game, video editor or second local model may already be using GPU memory. “8 GB installed” and “8 GB free” are different statements. A runner can still produce an answer when it splits the workload between GPU and system memory, but that does not mean the model fits entirely on the GPU. Ollama’s FAQ says its loaded-model display distinguishes 100% GPU, 100% CPU and a split percentage.

On an Apple-silicon Mac, the label changes again. Apple’s Metal technical talk says: “The CPU creates GPU resources within the unified memory block, and both the GPU and CPU are able to read and write to these resources.” A Mac with 32 GB of unified memory is not equivalent to a discrete card with 32 GB of dedicated VRAM: macOS, the CPU and other apps share that pool. Check the runner’s usable allocation and leave system headroom. The exact GPU-usable amount for a particular Mac and software version is unverified here.

Should I buy 8 GB, 16 GB or 24 GB for my own tasks?

For short chats, try a small model on your current computer. If you routinely paste codebases or long documents, prioritize headroom for context as well as model weights. A bigger card broadens the candidates you can test, but do not buy one solely because a model fits on paper. Response time and answer quality need a real trial.

On an 8 GB GPU, the listed 2.5 GB qwen3:4b and 5.2 GB qwen3:8b downloads offer reasonable starting experiments. Ask several normal questions, close GPU-heavy apps, and inspect the loaded-model split. The qwen3:4b page advertises a 256K-token ceiling, but that does not make such an enormous context a sensible setting for an 8 GB machine. If you are drafting a short letter, a smaller active window may be enough. The Ollama tag listing supplies sizes, not a guarantee of either speed or fit.

With 12–16 GB, a candidate such as the listed 9.3 GB qwen3:14b becomes less obviously squeezed, although you still have to inspect a real session. For coding assistance, use a representative file, a prompt with actual repository context and the sort of output you would request every day. A one-line “hello” test may hide the wait imposed by a much longer prompt. Whether that build fits entirely in 12 GB at your chosen settings remains unverified without a run.

At 24 GB, the listed 20 GB qwen3:32b is possible to investigate, not an automatic recommendation. The gap between download size and card capacity is narrow enough that context and buffers matter. If it spills into RAM, a shorter context or another quantized build may work better than forcing that tag. Someone aiming for the listed 43 GB llama3.1:70b should not expect its weights to fit wholly on one 24 GB card. Offloading or more than one GPU changes the configuration, not the capacity of that single card.

More memory may not fix the pause before the first word of a long-document response. Prompt processing and output generation are separate parts of a run. Ollama’s API usage documentation reports prompt_eval_duration separately from eval_duration. If a model reads a long PDF slowly but writes at a tolerable pace, its output tokens per second will not capture that initial wait.

An advertised 70B parameter count is also a poor shopping target by itself. A smaller model might do your particular job well, while a larger one might require a compromise in quantization or context. Before paying for a card, decide whether your complaint is weak answers, a memory error, or a long wait. Each points to a different experiment.

How do I tell whether my current GPU is enough?

Use the exact model tag, a realistic context window and two prompts you might actually send. Compare its published file size against free memory, then check whether the runner places the workload entirely on the GPU. Launching successfully is one test; getting a useful answer without an annoying wait on your real input is the other.

With Ollama, run ollama ps while the model is loaded. Its context guide shows SIZE, PROCESSOR and CONTEXT in that display. 100% GPU is the relevant result if your aim is all-GPU loading; a CPU/GPU split means it runs, but not entirely on the accelerator. Check the value after your long prompt, not just immediately after a tiny greeting. You should also watch the GPU memory used by other programs in your operating system’s monitoring tool.

Try one short everyday question, then a long input you are permitted to process locally. For a programmer, that could be a real function and its callers rather than a synthetic code snippet. For a student, it might be a chapter rather than a sentence. Record the exact tag, context setting and whether the model was fully GPU-loaded in each case. The two runs are deliberately different workloads, so their timings are not directly comparable benchmarks.

If the model spills into system RAM, try a smaller context or model before buying hardware. If it fits but still gives poor answers, another model may be the better experiment. If the output is fine yet the long prompt takes too long to process, inspect prompt-evaluation time rather than assuming unused VRAM would fix it. Ollama’s usage fields separate model load, prompt evaluation and response generation. For a repeatable comparison, change only one of model, context or hardware at a time.

GPU memory figures also need labels. A store listing’s GB, a runner’s GB and a system utility’s GiB do not always describe the same unit or the same available pool. It is easy to treat a near-fit as certain by subtracting rounded numbers. Keep a margin and trust the observed split and memory pressure over a neat subtraction.

FAQ: What else changes the VRAM answer for a local LLM?

A model can launch on a CPU yet respond slowly, or run mostly on a GPU until the chat grows too long. A model-size chart alone cannot tell you how usable either session will feel.

Can I run a local LLM without a dedicated GPU?

Yes. A CPU-capable runner can use system memory instead of dedicated VRAM, though a successful load does not tell you whether the response time will suit you. If you are comparing machines, do not count a CPU run as proof that a certain graphics card is enough. Ollama’s loaded-model display explicitly labels 100% CPU and 100% GPU as different placements.

Is 8 GB VRAM enough for a 7B or 8B model?

It can be enough for a suitably sized quantized build at a manageable context, but “8B” is not the whole specification. Ollama lists qwen3:8b at 5.2 GB, a download size rather than peak operating memory. On a laptop with several graphics-heavy apps open, inspect what is actually free and whether the session stays on the GPU.

Do two 12 GB GPUs act like one 24 GB card?

Not automatically. A runner must know how to split the workload, and each card has its own memory and connection to the other. The llama.cpp multi-GPU guide documents different split modes and notes that performance depends on the interconnect. Check support for your model and runner before shopping around an imagined combined pool.

Will more VRAM make generation faster?

It may help if the alternative is offloading work to system RAM. Extra capacity sitting unused, however, is not a speed dial. A different GPU, model architecture, quantization or prompt length can change the experience in other ways. Compare the delay while the prompt is read with the rate of subsequent output; Ollama reports those stages separately. No speed increase for a particular card is verified here.