Cluster
Local LLMs
Runners, UIs, and serving engines for open-weight models on your own machine. Same honest spine on every page. Hardware links below are disclosed Amazon Associates when present.
-
VRAM & RAM guide
Buyer-intent constraints for local LLMs: VRAM bands, offload, disk — disclosed hardware examples only.
-
Storage for local models
Buyer-intent NVMe/library disk notes for GGUF libraries — disclosed Associates only.
-
Ollama
Local runner for open-weight models. MIT on the app; model licenses vary.
-
LocalAI
MIT OpenAI-compatible local API / multi-backend engine — models still have their own licenses.
-
Jan
Open-source desktop ChatGPT-shaped app. Apache-2.0 with attribution request; models separate.
-
text-generation-webui
AGPL-3.0 local LLM UI/API (TextGen / oobabooga); model licenses still separate.
-
LM Studio
Desktop app for local LLMs. Free for personal/internal business; proprietary app.
-
Open WebUI
Self-hosted chat UI. Branding clause from v0.6.6 — not OSI open source on current mainline.
-
LibreChat
MIT ChatGPT-shaped multi-provider chat UI; self-host with API keys or local backends.
-
GPT4All
MIT desktop app for local open-weight chat; model licenses still separate.
-
llama.cpp
MIT C/C++ inference engine; GGUF / DIY path many wrappers sit near.
-
llamafile
Apache-2.0 single-file LLM runner (Cosmopolitan); model licenses separate.
-
KoboldCpp
Single-file GGUF/GGML runner + KoboldAI Lite UI. AGPL-3.0; download only from GitHub.
-
vLLM
Apache-2.0 high-throughput GPU serving — not a laptop chat toy.
-
SGLang
Apache-2.0 high-performance LLM/multimodal serving with RadixAttention — not a laptop chat toy.
-
mlx-lm
MIT text generation and fine-tuning on Apple silicon via MLX — not a CUDA stack.
-
LiteLLM
MIT-core OpenAI-compatible gateway to 100+ providers and local backends; enterprise/ is separate.
-
Continue
Apache-2.0 coding agent that can point at local models; acquired by Cursor.
Hardware that usually matters
Software here is mostly free or cheap. The box is not. Example Amazon listings — not rankings, not endorsements; prices and stock change. Same metal for Ollama and LM Studio — see Ollama and LM Studio for the software chooser.
- ASUS Prime GeForce RTX 5060 Ti 16GB — Mid-range 16 GB VRAM card; enough for many 7B–13B quantized models.
- GIGABYTE GeForce RTX 3090 24GB — Older generation; 24 GB VRAM still matters for larger local models.
- CORSAIR Vengeance DDR5 64GB (2×32GB) — System RAM when layers spill off the GPU or you CPU-offload.
- Samsung 990 PRO 2TB NVMe SSD — Model files eat disk; fast NVMe keeps pulls from feeling stuck.
Some links are Amazon Associates (tag=techgurunomic-20). We may earn a commission if you buy.
Free sample — Local LLM Chooser + Failure-Mode Sample: printable chooser and failure-mode worksheet (not a blank tease PDF). Tech-Gurunomics digital product (separate from Amazon Associates hardware links).
Paid worksheets pack ($39) — Local LLMs + Agents Without the Hype: checklists and failure modes we use on these pages. Tech-Gurunomics digital product (separate from Amazon Associates hardware links).
Paid worksheets pack ($29) — Local RAG Without the Hype — Eval & Failure-Mode Worksheets: retrieval/eval checklists and failure modes for local document chat. Tech-Gurunomics digital product (separate from Amazon Associates hardware links).