How to Run an LLM Locally: Hardware, Tools and Model Choices

How to run an LLM locally: pick hardware that fits the model, choose Ollama, LM Studio or llama.cpp, and follow the steps to your first model.

By Galactic News Editorial · Published 2026-11-07T15:04:06+00:00 · Updated 2026-11-07T15:04:06+00:00

You run an LLM locally by installing a runtime, downloading a quantized open-weight model that fits your memory, and chatting with it on your own machine. No cloud, no API key, and no internet after the first download. Your prompts stay on your computer. This guide covers the hardware, the three main tools, and the exact steps to get your first model running.

Three things decide whether local AI works for you: enough memory, the right runtime, and a model matched to your hardware. Get those right and a modern laptop runs a useful assistant in minutes. Get them wrong and you will fight out-of-memory errors all day.

What you need to run LLM locally

Memory is the single most important constraint. A practical rule is that a 4-bit quantized model uses roughly 0.6 to 0.7 GB of memory per billion parameters. A 7B model fits in about 5 to 6 GB. A 13B model needs about 9 to 10 GB. A 70B model needs about 40 GB. Add a few gigabytes of headroom for the operating system and the context window.

Quantization is what makes this possible on consumer hardware. It compresses model weights from 16-bit to 4-bit, cutting memory use by roughly four times with only a small quality loss. Most local models you download are already quantized in the GGUF format. The standard choice is Q4_K_M, which averages about 4.5 bits per weight and keeps quality close to the full model.

Your hardware checklist is short. You need 16 GB of system RAM to run 7B to 13B models well. Eight GB is enough for small 3B to 7B models. A GPU with 8 to 24 GB of VRAM gives the best speed, and an NVIDIA card is the easiest choice because CUDA support comes first in almost every local AI tool. Apple Silicon Macs are excellent too, because the GPU shares unified memory with the system. Any modern multi-core CPU works, and it handles tokenization plus any layers that spill out of VRAM. Keep 100 to 200 GB free on an NVMe SSD, since model files run 2 to 40 GB each and most people test several before settling on one.

Pick a runtime: Ollama, LM Studio, or llama.cpp

You need a runtime that loads the model and serves it. The three main choices share the same GGUF model files and the same llama.cpp engine underneath, so you can switch later without relearning anything.

Ollama is the best pick for developers. It is a command-line tool with a background service and a built-in API. You install it, pull a model, and run it. Its default local API listens on http://localhost:11434. It suits scripting, automation, and wiring models into your own apps. Ollama runs on Windows, macOS, and Linux.

LM Studio is the best pick for beginners. It is a desktop app with a graphical interface. You search for a model in the built-in catalog, hit download, and chat. No terminal and no config files. It shows whether a model fits in your VRAM before you download it. It exposes an OpenAI-compatible local server on http://localhost:1234/v1, so other tools can connect to it. LM Studio is free for home and work use. On Mac it also supports Apple's MLX framework, which squeezes more out of M-series chips.

llama.cpp is the best pick for maximum control. It is the engine under both Ollama and LM Studio, and running it directly gives you the newest model support and the leanest footprint. It is a command-line server with backends for Metal, CUDA, HIP, Vulkan, and SYCL. Use it when you want to tune GPU offloading, quantization, and server settings by hand.

Choose LM Studio if you want point-and-click. Choose Ollama if you want a clean command line with a built-in API. Choose llama.cpp if you want direct control of the engine.

How to run an LLM locally with Ollama

Ollama is the fastest route from zero to a working local model. Follow these steps in order.

1. Install Ollama. Download it from ollama.com for your operating system, or run the install script on Linux. Start the app or service when it finishes.
2. Check your hardware. Confirm how much RAM and VRAM you have free. Pick the largest model that fits at 4-bit quantization, using the table above.
3. Pull a model. Run ollama pull llama3.3:8b for a general 8B model, or ollama pull qwen2.5-coder:7b for a coding model. The download is about 5 GB.
4. Run it. Type ollama run llama3.3:8b. The model loads into memory and a chat prompt appears. Type your question and press Enter.
5. Check where it runs. In another terminal, run ollama ps. The Processor column shows whether the model runs on GPU, CPU, or a split. If it says CPU, check your GPU driver.
6. Use the API. Send a request to http://localhost:11434/api/generate from your own code. Any app that speaks HTTP can now use your local model.

By default, Ollama keeps a model in memory for 5 minutes after its last use, then unloads it. Set OLLAMA_KEEP_ALIVE=30m before starting the server to keep models loaded longer. Set it to -1 to keep a model loaded until you stop it. Unload a model by hand with ollama stop <model-name>.

If a model fails to load, the cause is usually memory. Pull a smaller quantization, such as the Q2_K build, or a smaller model. If the context window is the problem, set OLLAMA_CONTEXT_LENGTH to a smaller value before starting the server. The Ollama FAQ lists a 4,096-token default context window, so check your installed version when you tune this.

How to run an LLM locally with LM Studio

LM Studio skips the terminal entirely. The system requirements are modest: Windows 10 or 11 with an AVX2 CPU, macOS 14 or newer on Apple Silicon, or Linux on x64 or ARM64. LM Studio recommends at least 16 GB of RAM and 4 GB of dedicated VRAM on Windows. Macs need 16 GB or more of RAM for comfortable use, and 8 GB Macs should stick to smaller models and modest context sizes.

1. Install LM Studio. Download it from lmstudio.ai and install it like any desktop app.
2. Search for a model. Open the search tab and type a model name, for example llama-3.3-8b. LM Studio lists the available GGUF builds with a compatibility badge showing whether your hardware can run each one.
3. Pick the Q4_K_M build. It is the quantization LM Studio recommends by default. It keeps quality close to the full model while cutting file size by roughly four times.
4. Download and load. Click download, then load the model. LM Studio reads it into RAM and VRAM and reports a Ready state, usually within 10 to 30 seconds.
5. Chat. Type in the chat window. Use the slider to adjust how many model layers go to the GPU and how many stay in RAM. A 13B Q4 model on an 8 GB card can still run at 12 to 15 tokens per second with 85 to 90 percent of layers on the GPU, against 3 to 4 tokens per second on CPU alone.
6. Serve it locally. Start the local server to expose an OpenAI-compatible API on port 1234. Other apps can then call your local model like any hosted API.

Watch the context setting. Models load with a default context window read from the GGUF file's own metadata, often only 4,096 or 8,192 tokens. Raising context length increases RAM and VRAM use roughly in proportion. A 7B model at 4,096 context may use 6 GB total. The same model at 65,536 context can need 12 to 14 GB. If a model shows a red compatibility warning, the full-precision file will not fit. Pick a smaller parameter count or a more aggressive quantization.

How to run an LLM locally with llama.cpp

llama.cpp gives you the engine directly. It suits users who want the newest models first and full control over inference settings.

1. Get a GGUF model file. Download one from Hugging Face, or convert a model yourself with the tools in the llama.cpp repository.
2. Build or download llama.cpp. Prebuilt binaries exist for Windows, macOS, and Linux. Build from source if you need a specific GPU backend.
3. Start the server. Run llama-server -m model.gguf. The server listens on port 8080 in most builds. Add -ngl 99 to offload as many layers as possible to the GPU.
4. Chat in the browser. Open the server address in a browser. The built-in web interface gives you a chat window with sampling controls.
5. Tune as needed. Set the context size with -c, the GPU layers with -ngl, and the batch size to trade speed against memory.

llama.cpp is also the engine behind the other two tools, so what you learn here transfers. When a new model format appears, llama.cpp usually supports it first.

Choose your first model

Pick the largest open-weight model your memory can hold at 4-bit quantization. For most laptops that means a 7B to 14B model such as Llama 3.x, Qwen 2.5, Gemma 2, or Mistral. Good first picks are Llama 3.3 8B for general chat, Qwen 2.5 7B for coding and multilingual work, and Phi-4 14B for stronger reasoning in a small package.

Download the Instruct version of a model, not the base version. Instruct models are tuned to follow directions and answer questions. Base models complete text instead of answering, which feels broken in a chat window.

Test two or three models before you settle. Each one has a different style. One may be terse, another chatty. The right model is the one whose answers need the least editing for your work.

Privacy, cost, and limits

Local models keep your data on your machine. Prompts and documents never leave your computer. That matters for private notes, client work, and anything under a confidentiality rule. There are no per-token fees and no usage limits beyond your hardware.

The trade is capability. A 7B local model is weaker than a frontier cloud model at hard reasoning, long documents, and niche knowledge. It is strong at drafting, summarizing, coding help, and everyday questions. Use local models for private and routine work, and cloud models when the task needs frontier capability.

Local models also need your maintenance. You update the runtime, you download new models, and you fix your own errors. The troubleshooting is straightforward, and most problems are memory or driver issues with known fixes.

Bottom line

Running an LLM locally takes about fifteen minutes. Install Ollama or LM Studio, pull a 7B or 8B model in Q4_K_M, and start chatting. Match the model to your memory, keep a few gigabytes of headroom for context, and prefer the Instruct version. Your prompts stay private, there are no token fees, and you can switch models or tools at any time.