LiteLLM

What is LiteLLM?

LiteLLM is an open-source library and gateway that gives applications a unified, OpenAI-format interface to more than 100 large-language models. It supports providers such as OpenAI, Anthropic, Vertex AI, Bedrock, Ollama, Azure OpenAI, and others, reducing provider-specific integration work.

Developers can use the Python SDK directly in an application or deploy the self-hosted proxy as an organization-wide LLM gateway. The SDK normalizes responses and provider errors, while the router supports retry, fallback, and load balancing across deployments. The proxy adds virtual keys, budgets, centralized logging, guardrails, caching, and an administrative interface.

How to Use LiteLLM

1. Install the litellm package for direct SDK use or the proxy extra for the gateway.
2. Set credentials for the chosen model provider.
3. Call completion() with a provider-prefixed model name and OpenAI-style messages.
4. Add the router when requests need retries, fallback, or load balancing.
5. For centralized access, start the proxy and point an OpenAI-compatible client at its base URL.

Core Features

• Unified model interface: Calls more than 100 LLMs through a consistent OpenAI-style format.
• Python SDK: Supports completion, embeddings, image generation, streaming, and related endpoints.
• Routing: Provides retry, fallback, and load balancing across model deployments.
• Normalized errors: Maps provider failures to OpenAI-compatible exception types.
• Observability: Connects callbacks to tools such as Langfuse, MLflow, and Helicone.
• Self-hosted gateway: Adds virtual keys, budgets, logging, guardrails, caching, and administration.

Use Cases

• Multi-provider applications: Developers can switch or combine model vendors behind one calling pattern.
• Resilient inference: Platform teams can route around model failures or distribute traffic.
• Central LLM governance: Organizations can manage keys, budgets, logs, and guardrails through the proxy.
• Usage accounting: Teams can collect response costs and usage through callbacks.

Frequently Asked Questions

Is LiteLLM only a Python library?

No. It also includes a self-hosted, OpenAI-compatible proxy for centralized model access.

Can existing OpenAI clients use the gateway?

Yes. The documentation states that OpenAI-compatible clients can point to the LiteLLM proxy without changing their request format.