edgequake-litellm

edgequake-litellm is a LiteLLM-compatible Python package backed by the Rust edgequake-llm core. The intent is simple: keep the LiteLLM call shape, replace the Python network path with a native implementation, and preserve operational features such as streaming, tool calling, embeddings, and provider routing.

# Before
import litellm

# After
import edgequake_litellm as litellm

Install

pip install edgequake-litellm

Supported wheel targets:

The package uses abi3-py39, so one wheel per platform covers Python 3.9+.

Scope note: this package covers the LiteLLM-compatible chat and embedding API surface. The Rust crate also ships image-generation providers, but those APIs are not exposed through edgequake-litellm yet.

Quick Start

import asyncio
import edgequake_litellm as litellm

messages = [{"role": "user", "content": "Explain Rust ownership in one sentence."}]

# Sync
resp = litellm.completion("openai/gpt-4o-mini", messages, max_tokens=128)
print(resp.choices[0].message.content)

# Async
async def main() -> None:
    resp = await litellm.acompletion("anthropic/claude-3-5-haiku-20241022", messages)
    print(resp.content)

    stream = await litellm.acompletion("openai/gpt-4o-mini", messages, stream=True)
    async for chunk in stream:
        print(chunk.choices[0].delta.content or "", end="", flush=True)

asyncio.run(main())

Embeddings:

import edgequake_litellm as litellm

result = litellm.embedding(
    "openai/text-embedding-3-small",
    ["hello world", "rust is fast"],
)

print(result.data[0].embedding[:3])
print(len(result[0]))

Model Discovery

Programmatic model listing, capability filtering, and name/fuzzy search — backed by the Rust discovery engine:

import edgequake_litellm as litellm

# List providers (unified catalog — includes cohere, nvidia, etc.)
print(litellm.list_providers())

# Filter by capabilities (live discovery)
models = litellm.discovery.find_models(
    requires_vision=True,
    min_context_length=100_000,
    max_output_tokens=32_768,
)

# Offline capability search (no API keys)
static = litellm.discovery.find_static_models(requires_thinking=True)

# Search by name / fuzzy with input & output length bounds
hits = litellm.discovery.search_static_models_by_name(
    "claude sonnet",
    fuzzy=True,
    min_context_length=200_000,
    min_output_tokens=16_384,
)
for hit in hits:
    print(f"{hit.model.provider}/{hit.model.id} score={hit.score:.2f} ({hit.match_kind})")

# Exact lookup by ID or display name
model = litellm.discovery.lookup_model_by_name("openai", "GPT-4.1")

See docs/discovery.md for the full Rust + Python API reference.

Provider Routing

Pass provider/model as the model argument:

Embedding-only backend:

Supported Features

Application Attribution

Propagate caller identity to upstream providers (OpenAI OpenAI-Project, OpenRouter referer/title, Ollama X-Client-Request-Id, etc.) and OTEL spans.

import edgequake_litellm as eq
from edgequake_litellm import ApplicationContext, get_provider_attribution

# Per-call kwargs
eq.completion(
    "openrouter/anthropic/claude-3.5-sonnet",
    [{"role": "user", "content": "hi"}],
    application_id="my-backend",
    application_name="My Service",
    application_url="https://app.example.com",
    request_id="req-123",
)

# Reusable context
ctx = ApplicationContext(application_id="my-backend", request_id="req-456")
eq.completion("mock/test-model", [{"role": "user", "content": "hi"}], application_context=ctx)

# Catalog: full | passthrough | observability_only | none
assert get_provider_attribution("openai") == "full"
assert get_provider_attribution("ollama") == "passthrough"

Ingress from a web framework: ApplicationContext.from_headers(request.headers).

Defaults from env: EDGEQUAKE_APP_ID, EDGEQUAKE_APP_NAME, EDGEQUAKE_APP_URL, EDGEQUAKE_TENANT_ID.

See migration guide and observability.

Environment Setup

Module defaults:

import edgequake_litellm as litellm

litellm.set_default_provider("anthropic")
litellm.set_default_model("claude-3-5-haiku-20241022")

Environment defaults:

• LITELLM_EDGE_PROVIDER
• LITELLM_EDGE_MODEL
• LITELLM_EDGE_TIMEOUT
• LITELLM_EDGE_MAX_RETRIES
• LITELLM_EDGE_VERBOSE

LiteLLM Compatibility

Implemented:

• completion()
• acompletion()
• embedding()
• aembedding()
• stream=True on acompletion()
• stream() async generator
• response.choices[0].message.content
• response.to_dict()
• AuthenticationError, RateLimitError, NotFoundError, Timeout
• list_providers()
• detect_provider()
• discovery.discover_all() / adiscover_all()
• discovery.find_models() / afind_models()
• discovery.find_static_models()
• discovery.search_models() / search_static_models_by_name() / asearch_models()
• discovery.lookup_model_by_name()
• discovery.get_model_info("provider/model")

Behavior notes:

• synchronous streaming is intentionally not supported; use acompletion(..., stream=True) or stream()
• unsupported or extra keyword arguments are dropped for LiteLLM parity
• per-call api_key, api_base, and timeout parameters are accepted at the Python layer but not yet wired into the Rust core for every provider

Provider Examples

OpenAI-compatible custom gateway:

export OPENAI_COMPATIBLE_BASE_URL=https://api.groq.com/openai/v1
export OPENAI_COMPATIBLE_API_KEY=...

import edgequake_litellm as litellm

resp = litellm.completion(
    "openai-compatible/llama-3.3-70b-versatile",
    [{"role": "user", "content": "Write a one-line changelog summary."}],
)
print(resp.content)

Vertex AI:

export GOOGLE_CLOUD_PROJECT=my-project
gcloud auth application-default login

resp = litellm.completion(
    "vertexai/gemini-2.5-flash",
    [{"role": "user", "content": "Summarise this design review."}],
)

Jina embeddings:

import edgequake_litellm as litellm

vectors = litellm.embedding(
    "jina/jina-embeddings-v3",
    ["retrieval query", "retrieval document"],
)
print(len(vectors[0]))

Development

git clone https://github.com/raphaelmansuy/edgequake-llm.git
cd edgequake-llm/edgequake-litellm

python -m venv .venv
source .venv/bin/activate

pip install "maturin>=1.7" "pytest>=9.0.3" "pytest-asyncio>=0.24" "ruff>=0.3" "mypy>=1.8"
pip install . -v

pytest -q -k "not e2e"
ruff check python/
mypy python/edgequake_litellm --ignore-missing-imports

Release

Release tags are separate from the Rust crate:

• Rust crate: vX.Y.Z
• Python package: py-vX.Y.Z

Publish flow for edgequake-litellm:

1. bump edgequake-litellm/Cargo.toml
2. bump edgequake-litellm/pyproject.toml
3. update CHANGELOG.md
4. push the release-prep commit
5. wait for python-ci.yml to go green
6. push py-vX.Y.Z

python-publish.yml builds the sdist and wheels, smoke-tests the native wheels, publishes to PyPI, and can attach built artifacts to the GitHub Release.

Changelog

See CHANGELOG.md for the current release line and published history.

License

Apache-2.0. See ../LICENSE-APACHE.