Guide
VRAM and RAM for local LLMs
Size VRAM for the model class you will run daily; treat system RAM as overflow and NVMe as the library. ~7B Q4 often wants ~4–6 GB VRAM; ~13B Q4 commonly wants ~8–12 GB; ~70B Q4 weight footprint alone is often mid–high 30 GB+ before generous context. Offload saves an OOM when the model almost fits; it tanks tokens/sec when most layers live in CPU/RAM. This is not a “best GPU 2026” list — Tech-Gurunomics maps constraints, not thumbnails.
The software in the local LLMs cluster is mostly free or cheap. The box is not. Pair this page with Ollama, LM Studio, and llama.cpp — those are the runners; this is the metal they sit on. Treat bands below as planning estimates (quantized chat, single user), not warranties. Measure your exact GGUF/quant, context, and UI overhead.
Who it's for / who it's not
| For | Not for |
|---|---|
| Desktop builders sizing GPU → RAM → disk for private/open-weight chat | Thumbnail shoppers chasing “runs 70B” without checking quant file size |
| Users of Ollama / LM Studio / llama.cpp who want order-of-magnitude expectations | People who need frontier hosted APIs for strongest reasoning (different product) |
| Anyone planning capacity before buying another DIMM or card | Assuming Mac unified memory ≡ NVIDIA desktop VRAM bands |
Rough VRAM bands (quantized chat, single user)
People usually talk in parameter count (7B, 13B, 70B) plus a quant (often Q4-ish for “fits and still readable”). Ballparks for weights in VRAM — estimates as of 2026-09-10:
| Model class (Q4-ish chat) | Estimated VRAM for weights | Typical experience | Offload role |
|---|---|---|---|
| ~7B Q4 | ~4–6 GB (estimate) | Comfortable on many mid cards if context is modest | Rarely needed |
| ~13B Q4 | ~8–12 GB (estimate) | Sane on 8–16 GB cards until context/vision stack up | Bridge when almost fits |
| ~70B Q4 | mid–high 30 GB+ weights alone (estimate) | Full “all on GPU, long context, snappy” ≠ a 16 GB card | Common on 24 GB: partial GPU + RAM spill (usable, slower) |
Source framing: Tech-Gurunomics planning bands for quantized single-user chat — not vendor marketing charts. Bigger context, multimodal towers, speculative decoding, and concurrent chats all push up. Serving stacks like vLLM are a different job than solo Ollama chat.
VRAM vs system RAM vs disk
| Resource | What it decides | Planning rule of thumb |
|---|---|---|
| VRAM | Whether day-to-day chat feels pleasant | Buy to fit the model class you actually run |
| System RAM | How painful spill/offload is | 32–64 GB common “I might spill layers” enthusiast zone (estimate); 16 GB + full browser tabs ≠ a 70B plan |
| NVMe / disk | Whether you can keep a library without deleting every Tuesday | Fast NVMe helps load/switch/pull times — not magic tokens/sec once resident |
Pair with: Storage for local models.
GPU vs CPU offload
VRAM is where layers want to live for speed. System RAM is the overflow tank. Runners built around llama.cpp-style graphs (including what you meet via Ollama / LM Studio) can keep some layers on GPU and spill the rest to CPU/RAM.
- Offload helps when the model almost fits: you get an answer instead of an OOM, and the GPU still carries enough layers to not feel like 2018.
- Offload hurts when most of the model is in RAM/CPU. Tokens per second fall off a cliff. “It loads” ≠ “you will use it daily.”
- RAM size matters for spill. 16 GB system RAM that is already full of browser tabs is not a 70B offload plan. 32–64 GB is the common “I might spill layers” zone for enthusiasts; more if you keep many models and Docker UIs around.
If your goal is comfortable day-to-day local chat, buy toward fitting the model class you actually want in VRAM. Use offload as a bridge or for occasional bigger models — not as the headline feature on a sales thumbnail.
Disk: models are luggage
For a deeper buyer-intent pass on NVMe capacity vs peak MB/s, see storage for local models.
GGUF and other weight files are multi‑GB each. A library of a few 7B/13B quants is already tens of gigabytes; add 70B-class files and you are in “where did 500 GB go” territory fast. A fast NVMe mainly helps load and swap times, not magic tokens/sec once the model is resident. Slow or full disks make pulls and model switches feel broken when the GPU is fine.
Example hardware (disclosed Associates)
Same examples as the
local LLMs hub — real categories only (GPU / RAM / NVMe),
not endorsements, not a ranking. Prices and stock change. Tag
techgurunomic-20.
- ASUS Prime GeForce RTX 5060 Ti 16GB — mid-range 16 GB VRAM pattern: headroom for many 7B–13B quantized setups; not an honest “70B fully on-GPU” card.
- GIGABYTE GeForce RTX 3090 24GB — older generation, but 24 GB VRAM is still what a lot of people hunt when they want larger local models or less painful offload.
- CORSAIR Vengeance DDR5 64GB (2×32GB) — system RAM for layer spill, Docker UIs, browsers, and “I loaded two models to compare.”
- Samsung 990 PRO 2TB NVMe SSD — room for a real model library; faster pulls and swaps than a dying SATA drive.
Some links are Amazon Associates (tag=techgurunomic-20). Tech-Gurunomics may earn a commission if you buy.
Free sample — Local LLM Chooser + Failure-Mode Sample: printable chooser and failure-mode worksheet (not a blank tease PDF). Tech-Gurunomics digital product (separate from Amazon Associates hardware links).
Paid worksheets pack ($39) — Local LLMs + Agents Without the Hype: checklists and failure modes we use on these pages. Tech-Gurunomics digital product (separate from Amazon Associates hardware links).
Where the shopping advice falls over
- Vendor “GB required” charts use different quant names — always check the exact GGUF/quant file size you will load.
- Laptop dual-use GPUs share memory with the OS; desktop VRAM numbers do not transfer cleanly.
- Apple Unified Memory is a different architecture — do not paste NVIDIA VRAM bands onto a Mac and call it science.
- A strong GPU with a tiny SSD and 8 GB of RAM still feels cursed.
- “It loads” with heavy CPU offload ≠ “you will use it daily” (tokens/sec fall off a cliff).
- Serving stacks like vLLM are a different job than solo Ollama chat.
FAQ
How much VRAM do I need for a local 7B model?
Estimated: a ~7B Q4 chat setup is often comfortable in roughly 4–6 GB VRAM if context is modest and you are not stacking a heavy UI + embeddings on the same card. Measure your quant and context — Tech-Gurunomics treats this as a planning band, not a warranty (as of 2026-09-10).
How much VRAM for 13B vs 70B local LLMs?
Estimated: ~13B Q4 commonly wants on the order of 8–12 GB VRAM for a sane experience. ~70B Q4 weight footprint alone is often in the mid–high 30 GB+ range before generous context; a single 24 GB card usually means partial GPU residency + system RAM offload, a smaller quant, or a smaller model.
Is GPU offload to system RAM a good idea?
Yes as a bridge when the model almost fits (answer instead of OOM, GPU still carries enough layers). No as the headline plan when most of the model lives in RAM/CPU — tokens per second collapse. Prefer fitting the daily model class in VRAM.
How much system RAM for local LLMs?
Estimated: 32–64 GB is the common enthusiast “I might spill layers” zone; more if you keep many models and Docker UIs. 16 GB already full of browser tabs is not a 70B offload plan.
Does a fast NVMe make local LLMs generate tokens faster?
Mostly no once weights are resident in VRAM/RAM. A fast NVMe mainly helps first load, model switch, and pulls. Capacity still matters — see storage for local models.
Ollama or LM Studio — which hardware page applies?
Both. Runners sit on the same metal. Start with Ollama or LM Studio; use llama.cpp when you need engine knobs. Cluster hub: Local LLMs (includes Ollama vs LM Studio).
Verdict
Pick the model class you will actually run daily, size VRAM for that class first, then buy RAM for overflow and NVMe for the library. Use Ollama or LM Studio to get started; use llama.cpp when you need the engine knobs. More cluster context: Local LLMs. Stars and “runs 70B” thumbnails are not a shopping list. — Tech-Gurunomics, 2026-09-10.