LiteLLM in Python: One API for OpenAI, Anthropic, Bedrock, and Gemini with Cost Tracking (2026)

Updated: July 30, 2026

LiteLLM is an open-source Python SDK and self-hosted proxy that gives you a single OpenAI-compatible API for 100+ LLM providers (OpenAI, Anthropic, Bedrock, Vertex AI, Gemini, Azure, vLLM, Ollama, and more) with built-in cost tracking, retries, fallbacks, load balancing, caching, and per-key budgets. In practice that means one litellm.completion(...) call replaces four provider SDKs, and one YAML file replaces the reliability layer you'd otherwise write from scratch. This 2026 guide walks through the SDK, the proxy, cost callbacks, fallbacks, Redis caching, and the operational trade-offs I've hit shipping LiteLLM to production.

• LiteLLM 1.6x (2026) exposes 1,890+ models across 140+ providers behind one OpenAI-compatible interface, with a Rust core that reduces gateway overhead versus the earlier pure-Python path.
• Use the Python SDK (litellm.completion, acompletion, embedding) for in-process calls; use the proxy server for multi-tenant cost tracking, virtual keys, and shared caching.
• Fallbacks, retries, timeouts, and cooldowns are configured in router_settings. 429s send a deployment to cooldown immediately, protecting your latency budget.
• Cost tracking works out of the box for known models (response_cost in callback kwargs) and can be extended with input_cost_per_token/output_cost_per_token for self-hosted models.
• Redis semantic caching requires Redis Stack (RediSearch module) and redisvl. A plain managed Redis without RediSearch will fail to load the cache.
• LiteLLM is a gateway, not an orchestration framework. Pair it with LangGraph, DSPy, or your own code for prompts and agents.

What is LiteLLM and why use it?

LiteLLM is a Python library and proxy server built by BerriAI that translates OpenAI-style requests into whatever native format each provider expects, then normalizes the response back. If you've ever wired openai, anthropic, and boto3 into the same codebase, you already know the tax: three SDKs, three retry policies, three token-usage shapes, three cost formulas, and three places to break when a provider changes their API. LiteLLM collapses that surface into a single call signature.

Under the hood, LiteLLM in 2026 ships with a Rust core that handles request marshalling and connection pooling, with a thin Python surface on top. The library supports /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, and Anthropic's /messages, the same endpoints you'd otherwise hit provider by provider. The BerriAI/litellm GitHub repository is the authoritative source for supported providers and models; expect the list to shift monthly.

Two things make it worth adopting even if you use only one provider today. First, it decouples your business logic from provider names. Switching from openai/gpt-4o to anthropic/claude-sonnet-5 is a config change, not a refactor. Second, it gives you the reliability primitives (retries, cooldowns, fallbacks, and cost accounting) that you'd otherwise duplicate for each SDK. In my experience the second reason is what actually saves the on-call rotation.

When to reach for LiteLLM

• You call more than one provider from the same service (or plan to).
• You want per-team or per-key spend limits without writing your own metering.
• You need semantic or exact caching to cap cost-per-prediction on repeat queries.
• You self-host vLLM, TGI, or SGLang inference servers and want to expose them behind the same interface as the hosted APIs.

Installing LiteLLM and making your first cross-provider call

LiteLLM is one pip install. The proxy dependencies are an extra:

# SDK only (in-process calls, callbacks, router)
pip install "litellm==1.60.*"

# Proxy server (adds FastAPI, database, admin UI)
pip install "litellm[proxy]==1.60.*"

The minimum viable example is four lines. Set the provider API key as an environment variable, then call completion() with the provider prefix baked into the model name.

import os
from litellm import completion

os.environ["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY", "")
os.environ["ANTHROPIC_API_KEY"] = os.getenv("ANTHROPIC_API_KEY", "")

# OpenAI
resp = completion(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize LiteLLM in one sentence."}],
    max_tokens=80,
)
print(resp.choices[0].message.content)
print("cost:", resp._hidden_params["response_cost"])  # populated automatically

# Same code, different provider
resp = completion(
    model="anthropic/claude-haiku-4-5-20251001",
    messages=[{"role": "user", "content": "Summarize LiteLLM in one sentence."}],
    max_tokens=80,
)
print(resp.choices[0].message.content)

Two things to notice. The response object is OpenAI-shaped regardless of provider: choices[0].message.content, usage.prompt_tokens, usage.completion_tokens. And LiteLLM has already computed the dollar cost of the call and stashed it in _hidden_params["response_cost"], using its internal model_cost.json map. You get that for free on the very first call, with no callbacks configured.

Async, streaming, and embeddings

Async is acompletion, streaming is stream=True, embeddings is embedding(). Same normalization applies:

import asyncio
from litellm import acompletion, embedding

async def stream_example():
    stream = await acompletion(
        model="bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0",
        messages=[{"role": "user", "content": "Explain retries."}],
        stream=True,
    )
    async for chunk in stream:
        delta = chunk.choices[0].delta.content or ""
        print(delta, end="", flush=True)

asyncio.run(stream_example())

# Embeddings: same normalized shape for OpenAI, Cohere, Bedrock, VertexAI
emb = embedding(
    model="vertex_ai/text-embedding-005",
    input=["A production RAG pipeline needs a gateway."],
)
print(len(emb.data[0]["embedding"]))

Routing across OpenAI, Anthropic, Bedrock, and Gemini

Once you have more than one deployment for the same logical model, say, an Azure OpenAI in eastus2 and one in swedencentral, you graduate from completion() to litellm.Router. The Router assigns multiple physical deployments to a single logical model_name, then load-balances across them using one of six strategies: simple-shuffle, least-busy, usage-based-routing, usage-based-routing-v2, latency-based-routing, or cost-based-routing. The Router documentation recommends simple-shuffle as the production default because it avoids the per-request coordination overhead the smarter strategies require.

from litellm import Router

model_list = [
    # Two Azure deployments of the same model. LiteLLM will shuffle between them.
    {
        "model_name": "gpt-4o",  # logical name your app uses
        "litellm_params": {
            "model": "azure/gpt-4o-eastus2",
            "api_base": "https://eastus2.openai.azure.com/",
            "api_key": os.environ["AZURE_API_KEY_EASTUS2"],
            "api_version": "2024-10-21",
        },
        "tpm": 240_000,
    },
    {
        "model_name": "gpt-4o",
        "litellm_params": {
            "model": "azure/gpt-4o-sweden",
            "api_base": "https://sweden.openai.azure.com/",
            "api_key": os.environ["AZURE_API_KEY_SWEDEN"],
            "api_version": "2024-10-21",
        },
        "tpm": 240_000,
    },
    # Cross-provider fallback under a different logical name
    {
        "model_name": "gpt-4o-fallback",
        "litellm_params": {
            "model": "anthropic/claude-sonnet-5",
            "api_key": os.environ["ANTHROPIC_API_KEY"],
        },
    },
]

router = Router(
    model_list=model_list,
    routing_strategy="simple-shuffle",
    num_retries=2,
    timeout=30,
    fallbacks=[{"gpt-4o": ["gpt-4o-fallback"]}],
    # Share cooldown state across multiple app instances
    redis_host=os.environ["REDIS_HOST"],
    redis_port=6379,
    redis_password=os.environ.get("REDIS_PASSWORD"),
)

resp = router.completion(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Where did this call actually land?"}],
)
print(resp._hidden_params["custom_llm_provider"], resp.model)

The Router's real value shows up when you scale horizontally. If you run ten pods of your API service and each holds its own in-memory cooldown state, a Sweden outage will be re-discovered by every pod, wasting an entire retry cycle each. Pointing the Router at a shared Redis makes cooldowns a global fact: the first pod that trips the cooldown protects the other nine.

Fallbacks and retries for production reliability

Reliability in LiteLLM is layered. From narrowest to broadest: retries against the current deployment, ordered escalation across deployments in the same model_name group, and finally cross-group fallbacks to a totally different provider. The 2026 default is worth memorizing because it drives your P99. When a request gets a 429, the deployment goes on cooldown immediately, so subsequent calls skip it without the client waiting for another timeout.

model_list:
  - model_name: gpt-4
    litellm_params:
      model: azure/gpt-4-primary
      api_key: os.environ/AZURE_API_KEY
      order: 1   # tried first

  - model_name: gpt-4
    litellm_params:
      model: azure/gpt-4-secondary
      api_key: os.environ/AZURE_API_KEY_2
      order: 2   # used only when order=1 is unavailable

  - model_name: gpt-4-fallback
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2
  timeout: 30
  fallbacks:
    - gpt-4:
        - gpt-4-fallback
  enable_weighted_failover: true

Read that chain out loud: order-1 Azure primary, then order-2 Azure secondary, then OpenAI's public gpt-4o. With enable_weighted_failover: true, the Router exhausts other deployments in the same group before switching model groups, which matters when your only real problem was a bad region and both endpoints serve the same weights.

Context window exceeded fallbacks

A subtler failure is the "context window exceeded" error. The request is syntactically valid, but the model rejects it because prompt + expected completion overflows the window. LiteLLM handles this with a separate context_window_fallbacks list. Point it at a longer-context model:

router_settings:
  context_window_fallbacks:
    - gpt-4o:            # 128k window
        - anthropic/claude-sonnet-5   # 200k+ window

Any 400 with a context_length_exceeded code re-routes automatically without you needing to catch it in application code. Honestly, this alone has saved me from more one-off "why did that long PDF fail" tickets than I care to admit.

How does LiteLLM track cost per request?

LiteLLM ships with a bundled cost map, a JSON of per-token input and output prices for every model it supports, and computes response_cost on every call using the observed token counts. You don't need to wire this up. What you do need to wire up is where those numbers land: a database, a metrics pipeline, a Slack alert. That's what callbacks are for.

import litellm

def track_cost(kwargs, completion_response, start_time, end_time):
    call_id = kwargs.get("litellm_call_id")
    model = kwargs.get("model")
    cost = kwargs.get("response_cost", 0.0)  # USD
    latency_ms = int((end_time - start_time).total_seconds() * 1000)
    cache_hit = kwargs.get("cache_hit", False)
    metadata = kwargs.get("metadata", {}) or {}
    team = metadata.get("team", "unknown")

    # Push to your metrics store. Prometheus, StatsD, DuckDB, Postgres, pick one.
    print(
        f"[cost] call={call_id} team={team} model={model} "
        f"cost=${cost:.6f} latency_ms={latency_ms} cache_hit={cache_hit}"
    )

litellm.success_callback = [track_cost]
litellm.failure_callback = [track_cost]

resp = litellm.completion(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "hi"}],
    metadata={"team": "search-ranking", "env": "prod"},
)

The callback kwargs give you response_cost, cache_hit, model, messages, and any metadata you attached at call time. That last field is the whole ballgame for chargeback. Tag every call with the team, feature, and environment that made it, and you can bill correctly without any provider-side dashboard access.

Custom pricing for self-hosted and free models

If you serve your own model behind a vLLM endpoint, LiteLLM won't have a price for it. Set input_cost_per_token and output_cost_per_token in the deployment params, and set them to 0 explicitly if the model should bypass budget checks entirely (the custom pricing docs spell this out):

{
    "model_name": "llama-3-1-70b-internal",
    "litellm_params": {
        "model": "openai/meta-llama/Llama-3.1-70B-Instruct",  # OpenAI-compatible endpoint
        "api_base": "http://vllm-cluster.internal:8000/v1",
        "api_key": "sk-internal",
        "input_cost_per_token": 0,   # free, do not charge or budget
        "output_cost_per_token": 0,
    },
},

The LiteLLM proxy: self-hosted LLM gateway

Everything above assumed the SDK runs inside your application. That's fine for one service. Once you have three services calling LLMs, each with its own key rotation, retry config, and per-team budget logic, you'll want to hoist that concern into a shared proxy. LiteLLM's proxy is the same routing/cost/caching code you already saw, exposed as an OpenAI-compatible HTTP server that any OpenAI client can talk to.

# config.yaml (mount into the container at /app/config.yaml)
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: llama-3-internal
    litellm_params:
      model: openai/meta-llama/Llama-3.1-70B-Instruct
      api_base: http://vllm-cluster.internal:8000/v1
      api_key: sk-internal

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL   # PostgreSQL
  proxy_batch_write_at: 10                # batch spend rows every 10s

router_settings:
  routing_strategy: simple-shuffle
  redis_host: redis.internal
  redis_port: 6379

litellm_settings:
  drop_params: true
  set_verbose: false
  success_callback: ["prometheus"]        # scrape /metrics

# docker-compose.yml
services:
  litellm:
    image: ghcr.io/berriai/litellm:v1.60.0-stable
    command: --config /app/config.yaml --port 4000 --num_workers 4
    ports: ["4000:4000"]
    environment:
      - LITELLM_MASTER_KEY=sk-master-change-me
      - DATABASE_URL=postgresql://litellm:litellm@db:5432/litellm
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
    volumes:
      - ./config.yaml:/app/config.yaml
  db:
    image: postgres:16
    environment:
      - POSTGRES_USER=litellm
      - POSTGRES_PASSWORD=litellm
      - POSTGRES_DB=litellm

Once it's up, any application can hit it as if it were OpenAI. The critical operational feature is the virtual key: you mint a per-team API key at the admin endpoint, set a monthly budget, and hand it to that team. LiteLLM enforces the budget in the proxy, so a runaway loop in one service can't drain your OpenAI account.

# Mint a virtual key with a $200/month cap, scoped to two models
curl -X POST http://litellm:4000/key/generate \
  -H "Authorization: Bearer sk-master-change-me" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["gpt-4o", "claude-sonnet"],
    "max_budget": 200,
    "budget_duration": "30d",
    "team_id": "search-ranking",
    "tpm_limit": 100000,
    "rpm_limit": 200
  }'

# The team's app now calls the proxy instead of the provider
from openai import OpenAI

client = OpenAI(
    base_url="http://litellm:4000",
    api_key="sk-litellm-...", # the virtual key from above
)
r = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "budgets are boring but useful"}],
)

Zero code changes on the app side beyond swapping base_url. That's the whole appeal: teams keep using the OpenAI SDK they already know while operations owns the reliability, cost, and audit story centrally. Pair it with LLM observability tools like Langfuse or Arize Phoenix as a callback target and you'll have traces, spans, and spend in one dashboard.

Caching: exact-match, Redis, and semantic

Caching is where LiteLLM pays its rent on cost. There are three modes worth knowing.

Exact-match cache keys the request by a hash of model + messages + params. Same prompt, same params, cache hit. It's safe, cheap, and covers a surprising share of production traffic (retry storms, repeated tool-call arguments, duplicate cron jobs).

import litellm
from litellm.caching.caching import Cache

litellm.cache = Cache(type="redis", host="redis.internal", port=6379, ttl=3600)

# First call: hits provider, ~700 ms
r1 = litellm.completion(model="openai/gpt-4o-mini", messages=[{"role":"user","content":"ping"}])
# Second identical call: hits cache, ~2 ms, cost = 0
r2 = litellm.completion(model="openai/gpt-4o-mini", messages=[{"role":"user","content":"ping"}])
print(r2._hidden_params.get("cache_hit"))  # True

Redis semantic cache keys the request by embedding similarity. Two different phrasings of the same question can hit the same cache entry. This is where the cost savings really compound for a chatbot or a RAG frontend. But the operational cost is real, so read the warning below.

litellm.cache = Cache(
    type="redis-semantic",
    host=os.environ["REDIS_HOST"],
    port=int(os.environ["REDIS_PORT"]),
    password=os.environ.get("REDIS_PASSWORD"),
    similarity_threshold=0.85,   # tighter = fewer false hits
    ttl=600,
    redis_semantic_cache_embedding_model="openai/text-embedding-3-small",
)

S3 / GCS caching for batch and offline pipelines. Slower per read, but cheap and durable, and it survives Redis eviction.

LiteLLM vs LangChain: gateway vs orchestration

These two get compared constantly, but they solve different problems. LiteLLM is a gateway: it makes provider calls uniform, reliable, and observable. LangChain (and increasingly LangGraph) is an orchestration framework, turning those calls into chains, agents, tool loops, memory, and prompt templates. In production, most teams end up running both. LangChain writes to an OpenAI-compatible client, which points at the LiteLLM proxy, which routes to the actual model. If you need typed, validated outputs from an LLM, that concern also lives above LiteLLM, not inside it.

Concern	LiteLLM	LangChain
Primary role	Provider gateway / proxy	Prompt & agent orchestration
Unified API across providers	Yes, OpenAI-compatible	Yes, ChatModel wrappers
Retries, fallbacks, cooldowns	Built-in, per-deployment	Limited; expensive fallbacks common
Cost tracking per call	Automatic response_cost	Manual via callbacks
Virtual keys & budgets	Yes (proxy)	No
Semantic cache	Redis / Qdrant	Via LangChain cache abstraction
Chains / agents / tool loops	No (out of scope)	Core feature
Deployment shape	Library or standalone proxy	Library; LangServe for HTTP

My rule of thumb: if you asked "which model should this call go to and how much did it cost?" you want LiteLLM. If you asked "what is the sequence of steps this agent should take?" you want LangGraph or DSPy on top. They're complements, not competitors.

Production lessons: latency budgets and on-call

I've shipped LiteLLM in front of user-facing chat and background enrichment. A few things I'd tell my past self.

Watch the P99, not the median

The gateway adds a small amount of overhead per call: a few milliseconds for the SDK, tens of milliseconds for the proxy. That's invisible at the median and dominant at the tail once fallbacks kick in. Instrument the proxy's /metrics endpoint and alert on litellm_request_total_time P99, not average latency. A doubling of P99 usually means a provider is silently degrading and your retries are hiding it.

Cap num_retries and timeout

Repeating from the callout above: num_retries: 2, timeout: 30 is a sane starting point for interactive traffic. Batch jobs can go higher, but put them behind a queue where a slow attempt doesn't block a user.

Use a virtual key per team, not per environment

The temptation is to mint one prod key and one staging key. Don't. Mint one virtual key per (team, environment), and set budgets on the (team, prod) key that reflect what that team is allowed to spend. When someone's demo goes viral, you throttle only their key, not everyone's.

Don't treat semantic cache as free

Every semantic cache lookup runs an embedding, which itself costs money and time. If your traffic is largely unique (long RAG contexts, personalized prompts), you may spend more on embeddings than you save on cache hits. Measure the hit rate before enabling it broadly. A useful rule of thumb: if cache_hit=True is below 30% of calls after a week, turn semantic caching off and stick with exact-match.

Journal the config

The proxy's config.yaml is production infrastructure. Version it in git, review changes like any other Terraform. A silent edit to a fallbacks block can quietly change which provider serves 30% of your traffic, and the bill will show up two weeks later.

Frequently Asked Questions

Is LiteLLM production-ready?

Yes, with caveats. The SDK and proxy are used at scale by teams processing millions of requests per day. The known operational pitfalls are semantic-cache setup (needs Redis Stack), PostgreSQL connection exhaustion when running many proxy instances against one small database, and the temptation to over-retry. If you provision Postgres and Redis appropriately and cap retries, LiteLLM handles production load fine.

Does LiteLLM support streaming responses?

Yes. Pass stream=True to completion() or acompletion() and iterate the chunks. The shape matches OpenAI's streaming chat completions regardless of the underlying provider. The proxy also forwards streaming responses over Server-Sent Events, so OpenAI SDK clients pointed at the proxy get streaming for free.

How does LiteLLM handle rate limits and 429s?

When a deployment returns a 429, the Router immediately places it on cooldown so subsequent requests skip it without waiting for another timeout. If you configure a shared Redis, that cooldown is visible to every instance of your app or proxy, so a bad Azure region doesn't get re-discovered by every pod.

Can LiteLLM enforce per-team budgets?

Yes, through the proxy's virtual keys. Mint a key with max_budget and budget_duration, and the proxy blocks further requests once the cap is hit. You can also set tpm_limit and rpm_limit per key for rate control independent of dollar spend.

How much overhead does the LiteLLM proxy add?

Expect single-digit milliseconds for the SDK-only path and roughly 10 to 40 ms for an HTTP hop through the proxy inside a data center, dominated by TLS and PostgreSQL writes if you log spend synchronously. Use proxy_batch_write_at to batch spend writes and keep the write path off the request critical path.

Should I use LiteLLM if I only call one provider?

It's still worth it for the reliability primitives (cost tracking, retries, fallbacks, and caching) even against a single provider with multiple regions. If you're certain you'll never add a second provider and never need cost attribution, plain openai or anthropic is fine; otherwise the switching cost later is higher than adopting it now.