Local AI Hardware Guide: RAM, VRAM and Storage Rules of Thumb
Local AI hardware guide: VRAM, RAM, and storage rules of thumb. An 8B model needs about 5 GB at 4-bit; a 16 GB card is the sweet spot for 14B and 27B models.
By Galactic News Editorial · Published 2026-11-08T07:22:12+00:00 · Updated 2026-11-08T07:22:12+00:00
Running AI models on your own computer comes down to three numbers: VRAM, RAM, and storage. VRAM decides which models fit on your graphics card. RAM decides what runs on your CPU and on Apple Silicon. Storage decides how many models you can keep. This local AI hardware guide gives rules of thumb for all three, with real model sizes and real graphics card specs.
Local AI hardware guide: the one-table summary
| Memory budget | Model size that fits | Example at Q4_K_M | Typical hardware |
|---|---|---|---|
| 2-4 GB | 1B-3B | 3B model, about 2 GB | Old laptop, small edge box |
| 8 GB | 3B-8B | Llama 3.1 8B, about 5 GB | Mid-range laptop GPU, 16 GB Mac |
| 12 GB | 7B-14B | Qwen3 14B, about 9 GB | RTX 5070 |
| 16 GB | 7B-13B, or 27B tight | Gemma 3 27B, about 16 GB | RTX 5070 Ti, RTX 5080, 32 GB Mac |
| 24 GB+ | 13B-32B | 32B model, about 20 GB | RTX 4090, RTX 5090, workstation |
These are weights only. Leave headroom for the context cache and about 500 MB to 1 GB of overhead. A model that exactly fills your VRAM will not run well.
VRAM: the number that matters most
VRAM is the dedicated memory on your graphics card. When a model runs on the GPU, its weights live in VRAM. If the weights do not fit, the model either fails to load or spills to slower system RAM. That is why VRAM, not raw speed, is the first number to check.
Quantization decides how much VRAM a model needs. A full-precision model stores each weight as a 16-bit number. A 4-bit quantized model stores each weight in roughly 4 bits. The standard choice is Q4_K_M, which uses about 4.89 bits per weight. At that level, a good rule of thumb is about 0.6 GB of file per billion parameters. An 8 billion parameter model lands near 5 GB. A 14 billion parameter model lands near 9 GB.
Measured sizes confirm the rule. An 8B model at Q4_K_M measures about 4.58 GB. The same model at Q8_0 measures about 7.95 GB, and at Q6_K about 6.14 GB. At Q5_K_M it is about 5.33 GB. Going below 4-bit saves space but costs quality. At Q3_K_M, an 8B model is about 4 GB, and at Q2_K it is about 2.5 to 3.4 GB, with noticeable degradation. Community consensus is that Q4_K_M is the sweet spot, and quality falls off sharply below roughly 3 bits per weight.
Higher quantization levels buy quality when you have VRAM to spare. Q8_0 is near-lossless and fits small models on any desktop card. Q6_K is close to lossless in less space. For most people, Q4_K_M is the right default. Pick a bigger model at Q4_K_M over a smaller model at Q8_0 when you must choose.
RAM and unified memory
System RAM matters in two cases. First, when you run on CPU only, RAM is the budget. The model file must fit in RAM, plus the context cache. Second, on Apple Silicon, RAM is shared between the CPU and GPU. There is no separate VRAM. The total RAM is the budget for everything.
For CPU-only inference, small models work fine. Jan's own guide notes that CPU-only inference works for 3B to 7B models, with slower responses. Keep prompts short and close heavy apps. A laptop with 16 GB of RAM can run a 7B or 8B model on CPU at usable speed. It will not be fast, but it will work.
On Apple Silicon, the same math applies to unified memory. A Mac with 16 GB of unified memory can run 7B to 8B models comfortably. A Mac with 32 GB can run 13B-class models and larger. The MLX framework, used by LM Studio and Jan on Macs, is built for this shared-memory design. LM Studio's documentation recommends at least 16 GB of RAM as a starting point, and that figure matches what most guides suggest for 7B to 8B models.
A practical rule: buy RAM for the biggest model you plan to run, not the smallest. RAM is hard to upgrade later on most laptops. A 20B model such as gpt-oss:20b needs about a 14 GB download and runs on systems with 16 GB of memory. That leaves little room for anything else. A 32 GB machine is the comfortable floor for serious local work.
Storage: fast SSDs and how much space
Model files are large. A single 8B model at Q4_K_M is about 5 GB. A 14B model is about 9 GB. A 27B model is about 16 GB. A 70B model at low quantization is about 26 GB or more. If you keep five models, you need 25 to 80 GB of disk space for models alone. Add the apps, the operating system, and your other files, and 512 GB of total storage starts to feel tight.
Use an SSD, not a spinning hard drive. Model loading reads gigabytes of data at startup. On an SSD, an 8B model loads in seconds. On a hard drive, the same load takes far longer and the experience feels broken. NVMe SSDs are the best choice, but any SATA SSD is fine for loading models. The drive speed matters at load time, not during generation.
Keep at least 100 GB free for a local AI setup. That covers a handful of models plus room to download and test new ones. If you plan to keep large models, 1 TB of SSD storage is the comfortable choice. Models live in a cache folder on your system drive by default. Ollama stores models under a .ollama folder in your home directory. You can move that folder to a larger drive if your system drive fills up.
Picking a graphics card
For most people, the graphics card is the purchase that matters. The current NVIDIA lineup gives a clear ladder. The figures below are NVIDIA's official specifications and launch prices.
| Card | VRAM | Power draw | Launch price | Local AI fit |
|---|---|---|---|---|
| RTX 5050 | 8 GB GDDR6 | 130 W | $249 | 3B-8B models |
| RTX 5060 | 8 GB GDDR7 | 145 W | $299 | 3B-8B models |
| RTX 5060 Ti | 8 or 16 GB GDDR7 | 180 W | $379-$429 | 8B, or 14B with 16 GB |
| RTX 5070 | 12 GB GDDR7 | 250 W | $549 | 8B-14B models |
| RTX 5070 Ti | 16 GB GDDR7 | 300 W | $749 | 14B-27B models |
| RTX 5080 | 16 GB GDDR7 | 360 W | $999 | 14B-27B models |
| RTX 5090 | 32 GB GDDR7 | 575 W | $1,999 | 32B-70B models |
An 8 GB card is the entry point. It runs 3B to 8B models at Q4_K_M with room for context. That covers the most popular local models, including Llama 3.1 8B and Mistral 7B. A 12 GB card adds room for 14B models. A 16 GB card is the sweet spot for serious use: it fits 14B models easily and can hold a 27B model at 4-bit. A 32 GB card opens 32B and some 70B models at low quantization.
Check your power supply before you buy. The RTX 5070 draws 250 W, the 5070 Ti draws 300 W, and the 5090 draws 575 W. A card that your power supply cannot feed will crash under load. Check the card's physical size too. High-end cards are long and thick, and they do not fit every case.
Used cards are worth a look. An older card with 12 GB or 16 GB of VRAM can beat a new 8 GB card for local AI, even if it is slower at gaming. VRAM capacity matters more than the card's age for this job. A used RTX 3090 with 24 GB of VRAM remains a strong local AI card for exactly this reason.
The context cache: the hidden memory cost
Weights are only part of the memory bill. The context cache, also called the KV cache, grows with the length of the conversation. A short chat needs little cache. A long document, or a 128K-token context window, needs gigabytes of it.
Plan for the cache explicitly. If your model file is 5 GB and your card has 8 GB of VRAM, you have about 2 GB left for cache after overhead. That is enough for a normal chat. It is not enough for a 100-page document. If you work with long documents, either pick a smaller model or a card with more VRAM.
You can also shrink the context window. Most tools let you set it. Ollama defaults to a 4,096-token context window. Raising it costs memory. Lowering it saves memory. Set the window to match the job: short for chat, long for documents, and never larger than you need.
A step-by-step sizing checklist
Work through this list before you download anything.
- Pick the model first. Decide what you want to run: chat, coding, document work, or reasoning. A 7B or 8B model covers chat and general use.
- Check the file size. Look up the Q4_K_M file for that model. Multiply the parameter count by 0.6 GB for a quick estimate. An 8B model is about 5 GB.
- Add cache and overhead. Add 1 to 2 GB for the context cache and about 500 MB to 1 GB of overhead. An 8B model needs about 7 GB of memory in practice.
- Match it to your hardware. Compare that number with your VRAM, or your total RAM on Apple Silicon or CPU-only setups. If it does not fit, pick a smaller model or a lower quantization.
- Check storage. Make sure you have room for the download plus at least 100 GB free overall. Use an SSD.
- Test with a small model first. Download a 3B model, confirm the tool works, then move up. Do not start with a 70B download.
- Watch the power and heat. Local inference loads the GPU hard. Make sure the fans and the power supply can handle sustained use.
Bottom line
Local AI hardware is a memory problem, not a speed problem. The rule is simple: a model at Q4_K_M needs about 0.6 GB per billion parameters, plus 1 to 2 GB for cache and overhead. An 8 GB card runs the popular 8B models. A 16 GB card is the sweet spot for 14B and 27B models. A 32 GB machine handles the big ones. Buy memory first, keep your models on a fast SSD, and start small before you download the giant files.