Tool
llama.cpp
A lean C/C++ library and tooling to run LLM inference locally (CPU and GPU backends), famous for GGUF models and being the engine underneath a lot of friendlier wrappers.
What it is
llama.cpp (ggml-org) is open-source LLM inference in C/C++: load quantized weights, run prompts, serve locally. It is lower-level than Ollama or LM Studio — more knobs, more build/runtime choices, less “app store” polish. GitHub marks the project MIT License.
What it replaces
It replaces heavier Python stacks when you want portable local inference, and it replaces “I guess I need a full training framework to chat.” Wrappers (Ollama and others) often sit on related ideas or formats; llama.cpp is the DIY / embeddable path.
It does not replace picking a good model, enough RAM/VRAM, or reading the model license. MIT on the engine ≠ Meta/Gemma/Qwen terms on the weights.
License
The llama.cpp software is MIT (official LICENSE). Model files you download are separate legal objects — always check the model card.
Where it falls over
- Build flags, backends, and GPU drivers are a real onboarding tax versus one-click apps.
- Performance literacy matters: quantization, context length, and batching dominate UX.
- Server mode / tooling around it is powerful and easier to mis-expose on a network.
- Ecosystem forks and renames confuse “which binary am I running?”
Hardware that usually matters
The apps on these pages are free or free-to-use. The recurring cost is usually the machine: VRAM first, then system RAM, then disk for weights. Example Amazon listings skewed toward entry and mid budgets for beginners checking fit — not endorsements; stock and price move constantly.
- GIGABYTE GeForce RTX 3060 Eagle OC 12GB — entry 12 GB VRAM pattern many beginners still use for 7B–13B quantized chat before spending mid-range money.
- GIGABYTE GeForce RTX 4060 Ti AERO OC 16GB — mid 16 GB VRAM path when 12 GB starts feeling tight; still not an honest “70B fully on-GPU” card.
- EVGA GeForce RTX 3090 XC3 Ultra 24GB (Renewed) — older 24 GB class for larger local models / less painful offload; renewed stock and warranties differ from new.
- CORSAIR Vengeance LPX DDR4 32GB (2×16GB) — cheaper system RAM path for older DDR4 boxes when layers spill or you CPU-offload.
- Patriot Viper Venom DDR5 32GB (2×16GB) — DDR5 kit for newer boards; 32 GB is a common “I might spill layers” starting point.
- Samsung 990 EVO Plus 1TB NVMe SSD — entry NVMe for a first model library; step up when pulls fill the drive.
- Samsung 990 PRO 2TB NVMe SSD — default library disk for a mixed set of 7B/13B quants without deleting every Tuesday.
Some links are Amazon Associates. We may earn a commission if you buy.
Who it's for
Developers and tinkerers who want the inference engine itself, or who are debugging what a wrapper is doing underneath. Less ideal as a first local-LLM UI.
Verdict
Use llama.cpp when you want MIT-licensed, embeddable local inference and will own the build. Use Ollama/LM Studio when you want the batteries included — and still treat model licenses as separate.