Tool

LiteLLM

Local LLMs · LLM gateway / proxy · docs.litellm.ai

An OpenAI-shaped gateway that routes chat and completion calls across hosted providers and local backends (vLLM and friends), with optional cost tracking, load balancing, and guardrails — one SDK/proxy instead of twelve provider SDKs.

What it is

LiteLLM (BerriAI/litellm) is a Python SDK plus a proxy/gateway. You configure model aliases, point apps at an OpenAI-compatible endpoint, and LiteLLM translates to Bedrock, Azure, Anthropic, OpenAI, Vertex, vLLM, and many others. Teams use it as a single control plane for keys, spend, and failover — not as a model weights runner by itself.

What it replaces

It replaces hand-rolled multi-provider glue and “copy the OpenAI client six times” when you want one interface for cloud and local backends. It does not replace the inference box: for local models you still run Ollama / vLLM / llama.cpp (or a vendor API), and you still pay provider invoices for hosted traffic.

License

Per the project LICENSE: content outside the enterprise/ directory is available under the MIT License. Anything under enterprise/ uses a separate commercial license defined there. Do not call the whole product “pure MIT” if you depend on enterprise features — read both trees before you ship a commercial gateway.

Where it falls over

  • The gateway becomes a single point of failure if every app hard-depends on it.
  • You still own API keys, spend caps, and provider rate limits — LiteLLM routes; it does not print money.
  • Config and feature surface grow fast; mis-routed models waste money quietly.
  • Enterprise capabilities live under a separate license path — confirm what you actually deploy.
  • Local backends still need VRAM/ops; the proxy does not invent a GPU.

Who it's for

Teams and power users who want one OpenAI-compatible front door across providers and local servers, and who will operate Redis/DB/proxy configs. Wrong tool if you only needed a single vendor’s console, or a desktop chat app with no gateway ops.

Related in this cluster

  • vLLM — high-throughput local/GPU serving often behind LiteLLM
  • Ollama — local runner you can point a gateway at
  • LocalAI — OpenAI-compatible local API / multi-backend engine
  • Open WebUI — chat UI that can talk to OpenAI-shaped backends
  • VRAM & RAM guide — when the backend is local hardware

Verdict

Use LiteLLM when an MIT-licensed OpenAI-compatible gateway beats maintaining provider-specific clients — and you will own keys, spend, and the enterprise/ vs MIT split. Pair local routes with a real runner; the proxy is glue, not magic inference.

← Tools · Home · docs.litellm.ai · GitHub