Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

llm-queue

LLM calls through a queue on Valkey or Redis, under RPM / TPM budgets per deployment (model x account x region), balanced across regions and accounts, with fallback regions, fallback models, health checks and status. Works with Amazon Bedrock and OpenAI-compatible servers (vLLM), and with LangChain. The queue, the budgets and the results live in one store, so admission (deployment, RPM, TPM, concurrency slot) is one atomic step, and 1 pod behaves the same as 50.

Install

pip install "llm-queue[langchain-aws]"            # Bedrock + LangChain
pip install "llm-queue[openai,langchain-openai]"  # OpenAI-compatible
pip install "llm-queue[http]"                     # status and health routes for FastAPI

Example

from llm_queue import LLMQueue, CallFailed, request_context

llmq = LLMQueue.from_env()               # LLMQ_STORE_URL, LLMQ_REGISTRY, ...
await llmq.start()                       # publish / load the registry, run the workers
try:
    with request_context(priority=True, user_id=uid):
        reply = await llmq.converse("gpt120b", {"messages": [...]}, max_wait_secs=5)
    print(reply.response, reply.meta.deployment)
except CallFailed as exc:                # exc.kind: rate_limited, model_unavailable, ...
    ...
await llmq.stop()

Documentation

Full documentation: https://techiewonk.github.io/llm-queue/

  • Getting started: install, local stack, a registry, the first call
  • How-to guides: configure, tune, call, operate, share models across services, test
  • Explanation: architecture, the life of a call, admission, failures and recovery
  • Reference: settings, error kinds, CLI and HTTP, the Python API

Project: CHANGELOG, CONTRIBUTING, STABILITY, SECURITY.

Metadata

Release files for llm-queue 0.1.0b1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for llm-queue 0.1.0b1
File Size Uploaded Provenance Actions
llm_queue-0.1.0b1.tar.gz 135.3 kB With provenance Details for llm_queue-0.1.0b1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for llm-queue 0.1.0b1
File Interpreter ABI Platform Actions
llm_queue-0.1.0b1-py3-none-any.whl Python 3 none any Details for llm_queue-0.1.0b1-py3-none-any.whl

Total release size: 289.6 kB

Release files / llm_queue-0.1.0b1.tar.gz

Download URL llm_queue-0.1.0b1.tar.gz
Size 135.3 kB
Tags Source
SHA-256 checksum
How to use checksums
65a132bbbbf6b8b73902a1d3aec7149279d24e08086400feffaa54d66be0ecc1
BLAKE2b-256 checksum
How to use checksums
fa4d3e911a1bd4eec3e06b034dabdf5810deba48a5bc4cec5e8def9914c16a4c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Release files / llm_queue-0.1.0b1-py3-none-any.whl

Download URL llm_queue-0.1.0b1-py3-none-any.whl
Size 154.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
0b5168512a0028814e9475e33cff96682dcbd5efe96d744a24d570b2c501e445
BLAKE2b-256 checksum
How to use checksums
47f54e81f337d625598a02996b08c2dcbe0758ab575177ad28d52379d0aee32c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

Security for llm-queue 0.1.0b1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for llm_queue-0.1.0b1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release
Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page