SmolLLM

A minimal Python library for interacting with various LLM providers, featuring automatic API key load balancing and streaming responses.

Installation

pip install smolllm
uv add "smolllm @ ../smolllm"

Quick Start

import asyncio
from smolllm import ask_llm

async def main():
    response = await ask_llm(
        "Say hello world",
        model="gemini/gemini-2.0-flash"
    )
    print(response)

if __name__ == "__main__":
    asyncio.run(main())

Reasoning Effort

For reasoning-capable models, pass reasoning_effort through to the provider. Useful for Ollama reasoning models where omitting it can make replies much slower.

response = await ask_llm(
    "Reply with one word.",
    model="ollama/qwen3.5:0.8b",
    base_url="http://rocry-ubuntu.local:11434/v1",
    reasoning_effort="none",
)

Provider Configuration

Format: provider/model_name (e.g., openai/gpt-4, gemini/gemini-2.0-flash)

A bare model name (no /) targets an endpoint directly — pass base_url and api_key explicitly (no env fallback):

response = await ask_llm(
    "hi",
    model="gpt-4",
    base_url="https://my-proxy.example/v1",
    api_key="sk-xxx",
)

API Keys

The library looks for API keys in environment variables following the pattern: {PROVIDER}_API_KEY

Example:

# .env
OPENAI_API_KEY=sk-xxx
GEMINI_API_KEY=key1,key2  # Multiple keys supported

Key/endpoint pairs rejected with a permanent credential status (for example CONSUMER_SUSPENDED or API_KEY_INVALID) are removed from the active pool for the rest of the process lifetime. Error logs redact credential-shaped content; eviction is identified only by a one-way key fingerprint.

Custom Base URLs

Override default API endpoints using: {PROVIDER}_BASE_URL

Example:

OPENAI_BASE_URL=https://custom.openai.com/v1
OLLAMA_BASE_URL=http://localhost:11434/v1

Advanced Configuration

You can combine multiple keys and base URLs in several ways:

1. One key with multiple base URLs:

OLLAMA_API_KEY=ollama
OLLAMA_BASE_URL=http://localhost:11434/v1,http://other-server:11434/v1

2. Multiple keys with one base URL:

GEMINI_API_KEY=key1,key2
GEMINI_BASE_URL=https://api.gemini.com/v1

3. Paired keys and base URLs:

# Must have equal number of keys and URLs
# The library will randomly select matching pairs
GEMINI_API_KEY=key1,key2
GEMINI_BASE_URL=https://api.gemini.com/v1,https://api.gemini.com/v2

Environment Setup Best Practices

When using SmolLLM in your project, you should handle environment variables at your application level:

1. Create a .env file:

# .env
OPENAI_API_KEY=sk-xxx
GEMINI_API_KEY=xxx,xxx2
ANTHROPIC_API_KEY=sk-xxx

2. Load the file at the application/process boundary:

uv run --env-file .env app.py

Tips

• Keep sensitive API keys in .env (add to .gitignore)
• Create .env.example for documentation
• For production, consider using your platform's secret management system
• When using multiple keys, separate with commas (no spaces)