Best GPU for Running Llama Locally: VRAM Requirements by Model Size

The best GPU for Llama depends on VRAM: 8B fits 24 GB cards, 70B at 4-bit needs 48 GB, and 405B belongs in a data centre. Full requirements by size.

By Galactic News Editorial · Published 2026-11-07T19:31:53+00:00 · Updated 2026-11-07T19:31:53+00:00

Best GPU for Running Llama Locally: VRAM Requirements by Model Size

The best GPU for Llama is the one with enough VRAM for your model size and quantisation level. Everything else is secondary. Llama 3.1 comes in 8B, 70B and 405B sizes. Each size needs a very different memory budget. This guide works through the arithmetic first. Then it matches real graphics cards to each size. All card specifications come from vendor pages and datasheets.

How much VRAM a Llama model actually needs

Weights dominate the memory budget. At full half precision, every parameter takes two bytes. So an 8B model needs about 16 GB for weights alone. Quantisation shrinks that figure hard. A 4-bit GGUF build stores each weight in roughly half a byte. Ollama's default downloads show the real sizes. Its Llama 3.1 builds weigh 4.9 GB, 43 GB and 243 GB. Those are the 8B, 70B and 405B models at 4-bit.

Weights are not the whole story. The key-value cache grows with context length. All Llama 3.1 models support a 128K-token window. Hugging Face's launch post tabulates the cache cost. For the 405B it reaches about 123 GB at full context. Even the 8B needs headroom beyond its 4.9 GB of weights. Leave at least 20 percent spare VRAM. Long prompts will thank you.

The cards worth considering

Bandwidth matters after capacity. Token generation reads the full weights for every token. Faster memory means faster tokens, once the model fits. The RTX 5090 leads consumer cards on both measures. It pairs 32 GB with 1,792 GB/s of bandwidth. NVIDIA launched it in January 2025 at a $1,999 list price. Street prices have differed from list. Check current listings before budgeting.

Best GPU for Llama 8B: almost anything modern

The 8B model at 4-bit fits in 8 GB of VRAM. At half precision it needs a 24 GB card. An RTX 3090 or 4090 runs it at FP16 with room to spare. The same cards run the 8B at high quantisation with huge context headroom. A used RTX 3090 is the value pick here. It matches the 4090 on capacity at 24 GB. Its bandwidth is lower, so tokens arrive slower. For a single user, that trade usually makes sense.

Best GPU for Llama 70B: the 48 GB step

The 70B model changes the answer completely. Its 4-bit weights alone take 43 GB. No 24 GB or 32 GB card holds them. The RTX 6000 Ada is the single-card answer. Its 48 GB of ECC memory fits the model plus a working cache. It also draws only 300 W and fits dual-slot workstations. The alternative is two 24 GB cards in one machine. llama.cpp splits layers across both GPUs. Two used RTX 3090s cost less than one workstation card. They also draw 700 W and complicate cooling. At FP16, forget single machines below server class. The weights alone need about 140 GB.

Llama 405B: server territory

The 405B model is not a desktop project. Its 4-bit weights take 243 GB. Its FP16 weights take about 810 GB. Even eight 32 GB consumer cards barely cover the quantised build. Add the key-value cache at long context. The honest recommendation is a rented GPU cluster. Renting also skips the power and cooling problem. Run the 70B locally instead if you need privacy. It answers most questions nearly as well.

Software notes before you buy

1. Install Ollama for the fastest path to a running model.
2. Use llama.cpp directly when you need layer splitting across GPUs.
3. Download GGUF builds matched to your VRAM budget.
4. Start at 4-bit; raise precision only if quality suffers.
5. Watch VRAM during a long-context chat, not just at load.

Bottom line

Buy capacity first and bandwidth second. For Llama 8B, a 24 GB RTX 3090 or 4090 is the sweet spot. For Llama 70B at 4-bit, the 48 GB RTX 6000 Ada is the clean single-card pick. For everything the RTX 5090 covers, its 32 GB and high bandwidth make it the fastest consumer option. The 405B belongs in a data centre. This guide is research-based on vendor specifications, not firsthand benchmarks.

What about 8-bit and FP8 builds?

Four-bit is not the only quantisation worth knowing. Eight-bit builds keep quality very close to full precision. They cost about one byte per parameter. So the 8B model needs roughly 8.5 GB of weights. The 70B needs roughly 75 GB. That puts 8-bit 8B within reach of a 16 GB card. It keeps 8-bit 70B in multi-GPU territory. FP8 builds behave similarly on supported hardware. Blackwell and Hopper tensor cores accelerate FP8 natively. For most home users, 4-bit remains the practical default. The quality loss is small for chat workloads. The memory saving is enormous. Move up a level only when outputs disappoint. Test the same prompts at both settings first.

AMD cards and Apple Silicon

The RX 7900 XTX matches NVIDIA's 24 GB cards on capacity. Its 960 GB/s bandwidth is competitive on paper. The catch is software maturity. llama.cpp runs on AMD through its Vulkan and ROCm backends. Ollama supports ROCm on supported Linux setups. Expect more manual configuration than the NVIDIA path. Apple Silicon is a different trade again. Macs use unified memory shared with the CPU. Large-memory Mac configurations can hold big quantised models. Bandwidth is lower than a flagship NVIDIA card, though. Tokens arrive more slowly. A Mac suits quiet, always-on personal use. A discrete NVIDIA card suits speed. Choose based on which you value more.

A note on buying used RTX 3090s

The second-hand market made the RTX 3090 famous among local LLM users. It offers flagship capacity at a fraction of launch pricing. Two cautions apply before buying. Mining-era cards may have worn fans and tired thermal pads. Ask for a stress test before paying. Power draw is the second caution. A 3090 pulls 350 W under load. Two of them need a serious power supply. They also heat a small room quickly. Factor electricity and cooling into the bargain. A single newer card can cost less over two years. Run the numbers for your own usage pattern.