justllm

Production LLM calls. Just the three lines.

from justllm import LLM

llm = LLM("anthropic/claude-opus-4-8")
llm("Summarize this contract.")

That call already does the work you'd normally wire up yourself, on by default:

• Context compression. Headroom shrinks tool output by 50–95% before it reaches the model.
• Prompt-cache optimization. Cache breakpoints go where each provider wants them (Anthropic, OpenAI, Google).
• Reliability. Calls retry with backoff, then fail over to the next provider.

pip install 'justllm[all]'

A little more

Same three lines. Each of these is one call or one kwarg:

llm.extract(Invoice, text)                    # structured output (validated Pydantic)
llm.stream("...")                             # token streaming
await llm.acall("...")                        # async
llm.map(prompts, concurrency=8)               # many prompts at once, in order
llm.embed(texts)                              # embeddings
chat = llm.chat(); chat.send("..."); chat.send("...")   # multi-turn, remembers history
llm.agent(system="...").run("...")            # tool-calling loop
llm.judge(output, criteria="...")             # LLM-as-judge score
llm.evaluate(cases)                           # run + grade a test set
LLM(router=Cascade(small=cheap, large=big))   # cheap first, escalate when needed

A few more things sit behind opt-in extras: OpenTelemetry traces that include the per-call dollar cost (most setups leave that out), Langfuse-backed prompts, semantic cascade escalation, and exact-match caching. The hard parts are already wired; you just call them.

Runnable recipes: cookbook

Why

The ecosystem splits two ways. You can have powerful but heavy (LiteLLM, LangChain), or simple but thin (aisuite, any-llm). justllm sits in the middle: every optimization is on, and the surface stays at three lines. Keeping it that small was most of the work.

It runs on LiteLLM underneath, so think of it as the opinionated layer on top rather than a replacement.

Alpha. The wiring is tested on CI (Python 3.10–3.13) and the call paths are checked against live models.

Cookbook · Roadmap · Changelog · Contributing · MIT