flashlite 0.2.4

Batteries-included wrapper for litellm with rate limiting, retries, templating, and more

Flashlite

A batteries-included wrapper for litellm designed for high-volume prompting workloads like evals, agentic loops, and social simulations.

Features

• Rate Limiting - Token bucket algorithm for RPM and TPM limits
• Retries - Exponential backoff with jitter via tenacity
• Jinja Templating - Prompt templates with custom filters
• Caching - In-memory LRU and SQLite disk caching
• Structured Outputs - Native Pydantic model parsing without instructor
• Tool/Function Calling - @tool decorator and execution loops
• Multi-Turn Conversations - Conversation management with branching
• Cost Tracking - Token counting and budget limits
• Observability - Structured logging and Inspect framework integration
• Async-First - Native async with sync wrappers

Installation

pip install flashlite
# or with uv
uv add flashlite

Quick Start

from flashlite import Flashlite

# Create client (loads .env automatically)
client = Flashlite(default_model="gpt-4o")

# Simple completion
response = await client.complete(
    messages="What is the capital of France?"
)
print(response.content)

# Sync version
response = client.complete_sync(messages="Hello!")

Structured Outputs

from pydantic import BaseModel, Field
from flashlite import Flashlite

class Sentiment(BaseModel):
    label: str = Field(description="positive, negative, or neutral")
    confidence: float = Field(ge=0, le=1)

client = Flashlite(default_model="gpt-4o")

result: Sentiment = await client.complete(
    messages="Analyze: 'I love this product!'",
    response_model=Sentiment,
)
print(f"{result.label} ({result.confidence:.0%})")

Tool/Function Calling

from flashlite import Flashlite, tool, run_tool_loop

@tool()
def get_weather(location: str) -> str:
    """Get weather for a location."""
    return f"Weather in {location}: 72°F, sunny"

client = Flashlite(default_model="gpt-4o")

result = await run_tool_loop(
    client=client,
    messages=[{"role": "user", "content": "What's the weather in NYC?"}],
    tools=[get_weather],
)
print(result.content)

Parallel Processing

# Process many requests with concurrency control
responses = await client.complete_many(
    requests=[
        {"messages": f"Summarize: {doc}"}
        for doc in documents
    ],
    max_concurrency=10,
)

Caching

from flashlite import Flashlite, MemoryCache, DiskCache

# In-memory caching
client = Flashlite(
    cache=MemoryCache(max_size=1000),
    default_model="gpt-4o",
)

# Or persistent disk cache
client = Flashlite(
    cache=DiskCache("./cache.db"),
    default_model="gpt-4o",
)

Reasoning Models

from flashlite import Flashlite, thinking_enabled

client = Flashlite()

# OpenAI o1/o3
response = await client.complete(
    model="o3",
    messages="Solve this complex problem...",
    reasoning_effort="high",
)

# Anthropic Claude extended thinking
response = await client.complete(
    model="anthropic/claude-sonnet-4-5-20250929",
    messages="Complex reasoning task...",
    thinking=thinking_enabled(10000),
)

Documentation

• Quick Start Guide - Detailed examples for all features
• Development Guide - Setup and contributing
• Implementation Plan - Architecture and roadmap

Requirements

• Python 3.12+
• litellm
• pydantic>=2.0
• jinja2
• tenacity
• python-dotenv

License

MIT