open‑source Python package that lets you talk to many LLM providers through a single, consistent API
01What it is and how it works
LiteLLM sits between your application and the LLM providers (OpenAI, Anthropic, Cohere, etc.). You install the package, add a small config file that lists API keys and routing rules, and then replace direct SDK calls with litellm.completion(...). The library forwards the request, adds optional caching, logs usage, and returns the provider’s response in a uniform format. This abstraction saves you from writing separate client code for each vendor.
LiteLLM is a tool that makes it easy to use different AI models with the same code.
02What to do about it this week
Start by adding LiteLLM to a test project. Install the package, create a .env file with your API keys, and write a single completion call. Then enable the built‑in cost tracker and run a few prompts to see the usage report. Finally, set up a simple routing rule that sends high‑priority queries to a cheaper model and low‑priority ones to a more capable model.
- Install with
pip install litellm. - Store keys in environment variables, not in code.
- Define a
config.yamlthat maps model names to providers. - Run
litellm --track-costto generate a CSV of spend.
03How it is measured or noticed
LiteLLM writes a log entry for every request, including model name, token count, latency, and cost. You can watch the litellm.log file or enable the --verbose flag to see real‑time output. The cost tracker aggregates these entries into a daily summary that you can export to CSV or view in the console. Monitoring latency and error rates in the same log helps you spot provider outages quickly.
04Common mistakes
- Hard‑coding API keys in source files – they can be leaked.
- Skipping the
modelfield in the config – LiteLLM will default to the first provider, which may be expensive. - Disabling caching without a fallback – you may hit rate limits on cheap models.
- Assuming the response format is identical to the native SDK – some fields are renamed.
05Limits and confusion points
LiteLLM does not replace the underlying LLMs; it only forwards calls. It cannot run models that have no public API, such as on‑prem installations without a compatible endpoint. The library also does not handle fine‑tuning or custom tokenizers – those remain provider‑specific tasks. Users sometimes think LiteLLM is a model itself, but it is purely a routing and accounting layer.
06Worked example
`import litellm
response = litellm.completion(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Summarize the latest AI trends"}],
max_tokens=150
)
print(response.choices[0].message.content)`
In this snippet the same code would work forclaude-3-sonnet-20240229simply by changing themodelstring in the config. The log shows a $0.0005 cost and 0.23 s latency.
Frequently asked questions
How does LiteLLM differ from LangChain?
It depends on the layer you are looking at. LiteLLM is a thin wrapper that provides a unified API for calling many LLM providers, while LangChain adds higher‑level orchestration, prompting utilities, and chain building on top of LLM calls.
Should I use LiteLLM for production workloads, and what factors decide?
Usually you should adopt LiteLLM in production if you need consistent logging, cost tracking, and the ability to switch providers without code changes. The decision depends on your requirements for observability, provider diversity, and whether you can tolerate the extra dependency.
How do I integrate LiteLLM into an existing Python project?
Yes, you can integrate it by installing the package with pip, importing its client, and replacing your direct provider calls with LiteLLM’s completion or chat_completion methods. The wrapper forwards the request and automatically logs model name, token count, latency, and cost.
Does LiteLLM still work with the latest OpenAI API version?
Yes, LiteLLM is kept up‑to‑date with the official OpenAI SDK, so it continues to work with the current API version. If a breaking change occurs, the maintainers release a patch that you can upgrade to.
What happens if LiteLLM’s logs fail to capture token usage?
If logging fails, you will lose visibility into token counts and cost, which can lead to unexpected billing and difficulty debugging performance issues. You would notice missing entries in the log file or monitoring dashboard, prompting you to check the logging configuration.
How long does it take for LiteLLM’s cost metrics to appear after a request?
Usually the cost metrics are written to the log immediately after each request, but aggregated dashboards may need a few minutes to refresh. You can verify the entry directly in the log file to see the exact cost right away.
Asked out loud
spoken, not typedThe same term in the words somebody uses speaking to an assistant rather than typing into a box — written from the situation, which is why each one carries the situation it came from.
Yes, you can drop in LiteLLM as a thin wrapper; install it and point your existing calls to its client, and it will forward them to the provider without code changes.
Usually the higher cost comes from the token count logged by LiteLLM; check the log entries for model name and token usage to see if a larger model was called.
It depends; LiteLLM only forwards calls, so if the provider's API changed, you may need to update the provider config in LiteLLM, but the wrapper itself doesn't cause crashes.