Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

taskiq-llm-queue

Status: pre-alpha (0.1.0a1). Built and tested: the LLMQueue client, the SQS broker, adaptive limiter, claims, circuit breaker, retry and fallback, JSON and S3 result backends, Bedrock and OpenAI-compatible providers, LangChain/LangGraph adapters, canary routing, failover and the CLI. 619 tests on Valkey 8.1, ElasticMQ 1.7.1 and a fake LLM server; CI on Python 3.11–3.14. Breaking changes are still possible before 0.1 (see CHANGELOG).

An LLM request client with its own durable queue, built on taskiq and Amazon SQS.

Your code keeps calling LangChain models (ainvoke, aembed_query) or a small client API. Each call becomes a task on an SQS queue. Workers in your own app's processes run it under a limit learned from the provider's throttling, retry and fall back when needed, and return the result through Redis/Valkey.

your code ──> client / LangChain adapter ──> SQS (per model: -rt queue first, base, bulk)
                                               │
              result <── Redis/Valkey <── worker in your pods ──> Bedrock / OpenAI-compatible
            (per-pod dispatcher)          (adaptive limiter, retry, fallback)

Why

  • Spikes wait in a queue instead of failing. Throttled calls aren't lost or retried in a storm.
  • No quota numbers to maintain. The limit for each model and target is learned from throttle errors and shared by all pods.
  • Chat first. Separate -rt queues and reserved capacity keep live traffic ahead of background and bulk work.
  • No extra servers. The same app pods send and execute; SQS is only the broker.
  • Drop-in. LangChain/LangGraph code doesn't change: tool calling, callbacks (Langfuse, LangSmith) and response objects are identical.
  • Safe rollout. direct mode uses the same limiter, and a live canary moves traffic per model and lane without restarts.

Install

uv add "taskiq-llm-queue[langchain-aws]"                     # Bedrock + LangChain
uv add "taskiq-llm-queue[openai,langchain-openai]"           # OpenAI-compatible + LangChain
uv add "taskiq-llm-queue[speedups]"                          # hiredis parser for redis-py
Packages
Core taskiq, aiobotocore, redis>=5.0.1, pydantic>=2.10, orjson, structlog
openai openai, httpx
langchain-aws / langchain-openai LangChain 0.3 and 1.x are both supported
metrics prometheus-client
speedups hiredis

The package declares version ranges; pin exact versions in your app's lockfile. aiobotocore pins a narrow botocore window, so let your resolver pick an aiobotocore that matches your boto3.

Quickstart

from taskiq_llm_queue import LLMQueue, ModelConfig, QueueConfig, Target, request_context
from taskiq_llm_queue.langchain import QueuedChatBedrockConverse

llmq = LLMQueue(QueueConfig(
    key_prefix="llmq:local",
    redis_host="localhost", redis_port=6379, redis_password="localdev", redis_ssl=False,
    sqs_endpoint_url="http://localhost:9324",                 # ElasticMQ locally; omit on AWS
    models={
        "gpt120b": ModelConfig(
            provider="bedrock", model_id="openai.gpt-oss-120b-1:0",
            targets={"primary": Target(region="ap-south-1"),
                     "apac": Target(profile_id="apac.openai.gpt-oss-120b-1:0")},
            queues={"rt": "llm-gpt120b-rt", "default": "llm-gpt120b"}),
    },
))                                                            # or QueueConfig.from_env()  (LLMQ_* variables)

async def main():
    await llmq.start(mode="both")                             # "send" | "work" | "both"
    model = QueuedChatBedrockConverse(llmq=llmq, model_alias="gpt120b")
    with request_context(lane="rt", session_id="s-1", user_id="u-1"):
        reply = await model.ainvoke("Summarise today's market in one line.")
    print(reply.content, reply.response_metadata["served_model"])
    await llmq.stop()                                         # drains in-flight work

In a FastAPI app, call start() in the lifespan startup and stop() on shutdown. Send-only processes, such as CLIs and scripts, use mode="send".

Client API

Every call returns a result object with .meta: served_model, served_target, fallback_used, attempts, queue_wait_ms, provider_ms, worker.

res = await llmq.converse("gpt120b", {"messages": [...], "system": [...], "toolConfig": ...},
                          lane="rt", fallback="chain", timeout_secs=90, request_id="req-1")
res.response                    # exact Bedrock Converse dict

emb = await llmq.embed("titan", ["a", "b"], lane="bulk")
emb.vectors                     # list[list[float]]; `emb` itself also works as a list

chat = await llmq.chat("llama", {"messages": [{"role": "user", "content": "hi"}]})
chat.response                   # OpenAI-compatible chat completion dict

raw = await llmq.invoke_model("titan", {"inputText": "a"}, content_type="application/json")
raw.json(), raw.body, raw.content_type

await llmq.cancel("req-1")      # not run if not started; the caller gets LLMQueueCancelled
  • Timing: timeout_secs (relative) or deadline_at (absolute epoch seconds) on every call.
  • Model ids are aliases, optionally with a target: gpt120b, gpt120b@apac. Unknown aliases are rejected when the config is built.
  • Errors: everything derives from LLMQueueError: LLMQueueTimeout, LLMQueueCancelled, ProviderError(retryable, retry_after_secs), QueueUnavailable, ConfigError.
  • Sync callers: converse_sync, chat_sync, embed_sync, invoke_model_sync run direct under the shared limiter (counted as sync fallbacks). Set strict_async=True to raise instead. close_sync() (or stop()) closes the sync runner.
  • Jobs: register_job(name, fn) on workers, send_job(name, payload, timeout_secs=...) anywhere; jobs run on the job queues with the same claims and DLQ rules.

LangChain and LangGraph

Class Based on How
QueuedChatBedrockConverse langchain_aws.ChatBedrockConverse Builds the request and parses the reply with the installed langchain-aws's own code, so bind_tools, structured output, cache usage and service tier match exactly
QueuedBedrockEmbeddings langchain_aws.BedrockEmbeddings aembed_query / aembed_documents go through the queue
QueuedChatOpenAI langchain_openai.ChatOpenAI http_async_client with a queued httpx transport: chat completions and embeddings go through the queue; queue meta lands in response_metadata. The Responses API (/responses) is not supported yet
QueuedOpenAIEmbeddings langchain_openai.OpenAIEmbeddings Same transport; check_embedding_ctx_length=False by default for non-OpenAI servers
  • Model names: pass the queue alias as model_alias= (plus model_id= or llmq=). Passing an alias as model= raises a clear error.
  • Streaming: all queued classes force disable_streaming=True / streaming=False. astream() and LangGraph stream_mode="messages" return one complete message, so nothing bypasses the queue.
  • Laziness: queued classes find the running LLMQueue on first use and check routing and the kill switch on every call, so you can create them at import time.

Configuration

QueueConfig.from_env() reads LLMQ_* variables (durations end in _SECS; older _S names still work with a one-time warning). Notable settings:

Setting Default Note
aws_region None Left to the AWS credential chain / IRSA
redis_url or redis_host/port/username/password/ssl localhost redis_password is a SecretStr
redis_socket_timeout_secs / connect_timeout / health_check_interval 5 / 2 / 30 Connection errors retried 3× with backoff
id_hash_salt (LLMQ_ID_HASH_SALT) library constant Set your own in production; keys the BLAKE2b hash of session/user ids
timeout_secs per lane rt 90, bg 300, bulk 900
attempt_timeout_secs 180 One provider attempt; a hung target moves to the next fallback step
provider_read_timeout_secs / provider_connect_timeout_secs 300 / 10 Bedrock botocore and OpenAI httpx timeouts
sqs_send_timeout_secs 5 SQS counts as down only when no send completed for this long; then the call runs direct
throttle_retry_attempts 6 Tries per fallback step for throttles (backoff holds no slot)
valkey_wait_failover_secs 10 Result reads failing this long make waiting callers run direct
redis_max_connections 32 Raised automatically to worker slots + 16

Redis keys are cluster-safe (hash-tagged per request and per model+target), so cluster-mode Valkey (e.g. ElastiCache Serverless) works.

Providers

ModelConfig(provider="openai", model_id="llama-3.1-70b",
            targets={"primary": Target(base_url="http://vllm:8000/v1")}, api_key_env="LOCAL_LLM_KEY")
QueueConfig(allowed_providers={"bedrock"}, denied_hosts=["*.openai.azure.com"])   # enforced on sender and worker
QueueConfig(providers={"myprov": "mypkg.providers:MyProvider"})                    # custom provider, loaded by every pod

A custom provider implements async call(op, params, target) -> dict and classify(exc) -> ErrorInfo(kind="throttle" | "retryable" | "fatal", retry_after_secs). It can also be registered through the taskiq_llm_queue.providers entry point. SDK-level retries are off inside workers (botocore total_max_attempts=1, openai max_retries=0), so throttles reach the limiter.

Lanes and queues

Lane Queue Use
rt llm-<model>-rt Live user requests; polled first; may use the whole limit
bg llm-<model> Background work; at most 80 % of the limit
bulk llm-<model>-bulk if configured (e.g. embeddings), else base Ingestion/batch; lowest priority

A queued bg/bulk call waits at most 15 s for a slot, then goes back to the queue with a delay instead of holding a worker.

Results

Results are stored as JSON (never pickle) for 5 minutes and read once (GETDEL). Each pod runs one result dispatcher: a pub/sub wake reads only the announced results, and a periodic batched sweep catches the rest, so 10K waiting calls don't mean 10K polling loops.

Fallbacks: how to check and update

A fallback chain is tried inside the worker. Each step is retried with backoff before moving on:

  1. same model, primary target
  2. same model, another target (e.g. a cross-region profile gpt120b@apac)
  3. another model (only when the call's policy is chain)

Embeddings never switch model. Throttles shrink the limiter but never trip the circuit breaker; only 5xx and timeouts do.

uv run llm-queue fallback get                       # whole config (JSON); starts from your app's defaults
uv run llm-queue fallback get --model gpt120b       # resolved chain for one model
uv run llm-queue fallback test --model gpt120b      # dry run: which target would serve now
uv run llm-queue fallback set --model gpt120b --chain "gpt20b"
uv run llm-queue fallback set --file fallback.json  # merges per model; --replace to overwrite all
uv run llm-queue fallback reset                     # back to the startup default

Chains are validated and reloaded by every pod within 30 s with no restart. An invalid config is rejected and the last good one stays. Every change is written to an audit stream.

{ "chains": {
    "gpt120b": ["gpt20b"],
    "gpt20b": [{"model": "gpt120b", "require_headroom": true}]
  },
  "skip_targets": { "gpt120b": ["apac"] } }

To see fallbacks happen: res.meta / response_metadata show served_model, served_target, fallback_used and attempts, and the llmq_fallback_total{model,step} metric counts them. samples/05_fallback_on_throttle.py shows it end to end.

Operating it

uv run llm-queue routing set --model gpt20b --lane rt --pct 10   # canary: 10 % of sessions via SQS
uv run llm-queue routing kill on                                 # everything direct within 30 s
uv run llm-queue routing reset
uv run llm-queue limiter show --model gpt120b                    # learned limit, in-flight, pause
uv run llm-queue limiter reset --model gpt120b [--target apac]   # forget the learned limit
uv run llm-queue breaker show | reset
uv run llm-queue queues                                          # messages per queue and DLQ
uv run llm-queue job send ingest --payload-file payload.json --queue jobs-ingest

Failure behaviour

Failure What happens
Provider throttles Limiter shrinks and pauses that model + target; brokers stop receiving it unless another target in the chain has headroom; work waits in SQS
Provider down (5xx, timeouts) Circuit breaker opens → next fallback step
Worker killed mid-task Message reappears after the visibility timeout; the claim prevents a second provider call
Rolling deploy Pods stop receiving at once, release unstarted work, finish running calls, then exit (90 s budget in the reference deployment)
SQS unreachable That model switches to direct (+ shared limiter) for 300 s on transient send errors only. An ambiguous send is claimed first, so it can't run twice
Redis/Valkey unreachable Workers release messages instead of losing them; senders run direct with a local per-pod cap (last known limit ÷ live pods)

Logging

The library logs with structlog's key-value API on top of stdlib logging and never configures logging itself. Records are named taskiq_llm_queue.*; the message reads event key=value ... and the fields are also on record.llmq (a dict). Prompts, responses and params are never logged.

import logging
logging.getLogger("taskiq_llm_queue").setLevel(logging.INFO)   # stdlib apps: that's all

# loguru apps: the usual InterceptHandler picks these up like any stdlib logger.
# structlog apps: use structlog.stdlib.ProcessorFormatter on your handler to render them.

The llm-queue CLI logs warnings and errors to stderr, so its stdout is always plain JSON.

Security and data

  • Encryption is server-side on the queues (SSE-SQS or SSE-KMS); the library needs no setting for either.
  • Prompts and responses are never logged. user_id / session_id are hashed (keyed BLAKE2b) in task labels; raw ids stay inside the message body.
  • Nothing secret goes into messages; credentials come from the AWS chain (IRSA / Pod Identity) or env.
  • Unknown task names or newer message versions are released for another pod, never silently dropped.

Run it locally (no AWS needed)

make up              # Valkey (password) + ElasticMQ + fakellm
make test            # unit + integration against the stack
make down
# Windows without make:  uv run python tools/llmq_dev.py up | test | down

Runnable examples are in samples/: quickstart, chat, embeddings, lanes, fallback on throttle, send-only and worker pods, LangChain, LangGraph, jobs, cancel and timeouts, sync calls, CLI, and 13_research_assistant_flow.py, which replays a real app's flows locally or against real Bedrock (--target aws, settings from .env or AWS Secrets Manager). The stack, capability checks and test layers are described in docs/TESTING.md; delivery-safety rules in docs/DESIGN.md.

Credits

Broker design and feature set build on taskiq-aio-sqs (MIT, © 2022-2024 Jesse Constante) and taskiq-sqs (MIT, © 2026 Mike Shultz and the Taskiq team). Licence: MIT.

Metadata

Release files for taskiq-llm-queue 0.1.0a1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for taskiq-llm-queue 0.1.0a1
File Size Uploaded Provenance Actions
taskiq_llm_queue-0.1.0a1.tar.gz 165.5 kB With provenance Details for taskiq_llm_queue-0.1.0a1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for taskiq-llm-queue 0.1.0a1
File Interpreter ABI Platform Actions
taskiq_llm_queue-0.1.0a1-py3-none-any.whl Python 3 none any Details for taskiq_llm_queue-0.1.0a1-py3-none-any.whl

Total release size: 345.4 kB

Release files / taskiq_llm_queue-0.1.0a1.tar.gz

Download URL taskiq_llm_queue-0.1.0a1.tar.gz
Size 165.5 kB
Tags Source
SHA-256 checksum
How to use checksums
fa7942009dca099f40daa9e472dc3f90c7e9edddc49a4928448397a7513bd6ab
BLAKE2b-256 checksum
How to use checksums
6dbcdb3dc2fccc77aa04ddbea165a42ffaa37f2f6a8482a011202ad349df0845
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 29, 2026.

Transparency log

Release files / taskiq_llm_queue-0.1.0a1-py3-none-any.whl

Download URL taskiq_llm_queue-0.1.0a1-py3-none-any.whl
Size 179.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
865e08eb76fba5a7482464e93a488f7fd70bd845314d2da8a8af7bdfc1f1a492
BLAKE2b-256 checksum
How to use checksums
d0f489281104fb38fd5250a7af10df27eed8225edd9d01e790e94ff2ee93a3f4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 29, 2026.

Transparency log

Security for taskiq-llm-queue 0.1.0a1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for taskiq_llm_queue-0.1.0a1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 29, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release
Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page