Tool
vLLM
An open-source library for fast LLM inference and serving, aimed at throughput and concurrent requests on serious GPUs — not a beginner chat app.
What it is
vLLM is an inference and serving engine for large language models (OpenAI-compatible style APIs, continuous batching, PagedAttention lineage, etc.). Docs describe origins in UC Berkeley’s Sky Computing Lab and a large multi-org contributor base. GitHub lists Apache License 2.0.
What it replaces
It replaces naive “one request at a time” serving when you need higher GPU utilization for many users or parallel jobs. For teams, it can replace ad-hoc Flask wrappers around Hugging Face generate calls.
It does not replace Ollama for “chat on my MacBook.” If you do not have the hardware and ops appetite, vLLM is the wrong layer.
License
Apache License 2.0 (vllm-project/vllm). Model weights remain under their own terms.
Where it falls over
- Hardware bar is real: VRAM, drivers, and CUDA/ROCm reality checks.
- Ops surface: versions, model formats, and parallel settings fail loudly.
- Overkill and frustrating on consumer laptops; designed for serving, not toy demos.
- Security: an open OpenAI-compatible port on the public internet is an invitation.
Hardware that usually matters
The apps on these pages are free or free-to-use. The recurring cost is usually the machine: VRAM first, then system RAM, then disk for weights. Example Amazon listings skewed toward entry and mid budgets for beginners checking fit — not endorsements; stock and price move constantly.
- GIGABYTE GeForce RTX 3060 Eagle OC 12GB — entry 12 GB VRAM pattern many beginners still use for 7B–13B quantized chat before spending mid-range money.
- GIGABYTE GeForce RTX 4060 Ti AERO OC 16GB — mid 16 GB VRAM path when 12 GB starts feeling tight; still not an honest “70B fully on-GPU” card.
- EVGA GeForce RTX 3090 XC3 Ultra 24GB (Renewed) — older 24 GB class for larger local models / less painful offload; renewed stock and warranties differ from new.
- CORSAIR Vengeance LPX DDR4 32GB (2×16GB) — cheaper system RAM path for older DDR4 boxes when layers spill or you CPU-offload.
- Patriot Viper Venom DDR5 32GB (2×16GB) — DDR5 kit for newer boards; 32 GB is a common “I might spill layers” starting point.
- Samsung 990 EVO Plus 1TB NVMe SSD — entry NVMe for a first model library; step up when pulls fill the drive.
- Samsung 990 PRO 2TB NVMe SSD — default library disk for a mixed set of 7B/13B quants without deleting every Tuesday.
Some links are Amazon Associates. We may earn a commission if you buy.
Who it's for
People serving open models to multiple clients on GPUs they actually have. Researchers and platform teams, not first-week local-LLM tourists.
Verdict
Use vLLM when Apache-2.0 high-throughput serving matches your GPU box. Use Ollama/llama.cpp when you want personal local chat with less serving machinery.