This release is a pre-release and may not be stable for production use.
taskiq-llm-queue
Status: pre-alpha (
0.1.0a1). Built and tested: theLLMQueueclient, the SQS broker, adaptive limiter, claims, circuit breaker, retry and fallback, JSON and S3 result backends, Bedrock and OpenAI-compatible providers, LangChain/LangGraph adapters, canary routing, failover and the CLI. 619 tests on Valkey 8.1, ElasticMQ 1.7.1 and a fake LLM server; CI on Python 3.11–3.14. Breaking changes are still possible before 0.1 (see CHANGELOG).
An LLM request client with its own durable queue, built on taskiq and Amazon SQS.
Your code keeps calling LangChain models (ainvoke, aembed_query) or a small client API. Each call becomes a task on an SQS queue. Workers in your own app's processes run it under a limit learned from the provider's throttling, retry and fall back when needed, and return the result through Redis/Valkey.
your code ──> client / LangChain adapter ──> SQS (per model: -rt queue first, base, bulk)
│
result <── Redis/Valkey <── worker in your pods ──> Bedrock / OpenAI-compatible
(per-pod dispatcher) (adaptive limiter, retry, fallback)
Why
- Spikes wait in a queue instead of failing. Throttled calls aren't lost or retried in a storm.
- No quota numbers to maintain. The limit for each model and target is learned from throttle errors and shared by all pods.
- Chat first. Separate
-rtqueues and reserved capacity keep live traffic ahead of background and bulk work. - No extra servers. The same app pods send and execute; SQS is only the broker.
- Drop-in. LangChain/LangGraph code doesn't change: tool calling, callbacks (Langfuse, LangSmith) and response objects are identical.
- Safe rollout.
directmode uses the same limiter, and a live canary moves traffic per model and lane without restarts.
Install
uv add "taskiq-llm-queue[langchain-aws]" # Bedrock + LangChain
uv add "taskiq-llm-queue[openai,langchain-openai]" # OpenAI-compatible + LangChain
uv add "taskiq-llm-queue[speedups]" # hiredis parser for redis-py
| Packages | |
|---|---|
| Core | taskiq, aiobotocore, redis>=5.0.1, pydantic>=2.10, orjson, structlog |
openai |
openai, httpx |
langchain-aws / langchain-openai |
LangChain 0.3 and 1.x are both supported |
metrics |
prometheus-client |
speedups |
hiredis |
The package declares version ranges; pin exact versions in your app's lockfile. aiobotocore pins a narrow botocore window, so let your resolver pick an aiobotocore that matches your boto3.
Quickstart
from taskiq_llm_queue import LLMQueue, ModelConfig, QueueConfig, Target, request_context
from taskiq_llm_queue.langchain import QueuedChatBedrockConverse
llmq = LLMQueue(QueueConfig(
key_prefix="llmq:local",
redis_host="localhost", redis_port=6379, redis_password="localdev", redis_ssl=False,
sqs_endpoint_url="http://localhost:9324", # ElasticMQ locally; omit on AWS
models={
"gpt120b": ModelConfig(
provider="bedrock", model_id="openai.gpt-oss-120b-1:0",
targets={"primary": Target(region="ap-south-1"),
"apac": Target(profile_id="apac.openai.gpt-oss-120b-1:0")},
queues={"rt": "llm-gpt120b-rt", "default": "llm-gpt120b"}),
},
)) # or QueueConfig.from_env() (LLMQ_* variables)
async def main():
await llmq.start(mode="both") # "send" | "work" | "both"
model = QueuedChatBedrockConverse(llmq=llmq, model_alias="gpt120b")
with request_context(lane="rt", session_id="s-1", user_id="u-1"):
reply = await model.ainvoke("Summarise today's market in one line.")
print(reply.content, reply.response_metadata["served_model"])
await llmq.stop() # drains in-flight work
In a FastAPI app, call start() in the lifespan startup and stop() on shutdown. Send-only processes, such as CLIs and scripts, use mode="send".
Client API
Every call returns a result object with .meta: served_model, served_target, fallback_used, attempts, queue_wait_ms, provider_ms, worker.
res = await llmq.converse("gpt120b", {"messages": [...], "system": [...], "toolConfig": ...},
lane="rt", fallback="chain", timeout_secs=90, request_id="req-1")
res.response # exact Bedrock Converse dict
emb = await llmq.embed("titan", ["a", "b"], lane="bulk")
emb.vectors # list[list[float]]; `emb` itself also works as a list
chat = await llmq.chat("llama", {"messages": [{"role": "user", "content": "hi"}]})
chat.response # OpenAI-compatible chat completion dict
raw = await llmq.invoke_model("titan", {"inputText": "a"}, content_type="application/json")
raw.json(), raw.body, raw.content_type
await llmq.cancel("req-1") # not run if not started; the caller gets LLMQueueCancelled
- Timing:
timeout_secs(relative) ordeadline_at(absolute epoch seconds) on every call. - Model ids are aliases, optionally with a target:
gpt120b,gpt120b@apac. Unknown aliases are rejected when the config is built. - Errors: everything derives from
LLMQueueError:LLMQueueTimeout,LLMQueueCancelled,ProviderError(retryable, retry_after_secs),QueueUnavailable,ConfigError. - Sync callers:
converse_sync,chat_sync,embed_sync,invoke_model_syncrundirectunder the shared limiter (counted as sync fallbacks). Setstrict_async=Trueto raise instead.close_sync()(orstop()) closes the sync runner. - Jobs:
register_job(name, fn)on workers,send_job(name, payload, timeout_secs=...)anywhere; jobs run on the job queues with the same claims and DLQ rules.
LangChain and LangGraph
| Class | Based on | How |
|---|---|---|
QueuedChatBedrockConverse |
langchain_aws.ChatBedrockConverse |
Builds the request and parses the reply with the installed langchain-aws's own code, so bind_tools, structured output, cache usage and service tier match exactly |
QueuedBedrockEmbeddings |
langchain_aws.BedrockEmbeddings |
aembed_query / aembed_documents go through the queue |
QueuedChatOpenAI |
langchain_openai.ChatOpenAI |
http_async_client with a queued httpx transport: chat completions and embeddings go through the queue; queue meta lands in response_metadata. The Responses API (/responses) is not supported yet |
QueuedOpenAIEmbeddings |
langchain_openai.OpenAIEmbeddings |
Same transport; check_embedding_ctx_length=False by default for non-OpenAI servers |
- Model names: pass the queue alias as
model_alias=(plusmodel_id=orllmq=). Passing an alias asmodel=raises a clear error. - Streaming: all queued classes force
disable_streaming=True/streaming=False.astream()and LangGraphstream_mode="messages"return one complete message, so nothing bypasses the queue. - Laziness: queued classes find the running
LLMQueueon first use and check routing and the kill switch on every call, so you can create them at import time.
Configuration
QueueConfig.from_env() reads LLMQ_* variables (durations end in _SECS; older _S names still work with a one-time warning). Notable settings:
| Setting | Default | Note |
|---|---|---|
aws_region |
None |
Left to the AWS credential chain / IRSA |
redis_url or redis_host/port/username/password/ssl |
localhost | redis_password is a SecretStr |
redis_socket_timeout_secs / connect_timeout / health_check_interval |
5 / 2 / 30 | Connection errors retried 3× with backoff |
id_hash_salt (LLMQ_ID_HASH_SALT) |
library constant | Set your own in production; keys the BLAKE2b hash of session/user ids |
timeout_secs per lane |
rt 90, bg 300, bulk 900 | |
attempt_timeout_secs |
180 | One provider attempt; a hung target moves to the next fallback step |
provider_read_timeout_secs / provider_connect_timeout_secs |
300 / 10 | Bedrock botocore and OpenAI httpx timeouts |
sqs_send_timeout_secs |
5 | SQS counts as down only when no send completed for this long; then the call runs direct |
throttle_retry_attempts |
6 | Tries per fallback step for throttles (backoff holds no slot) |
valkey_wait_failover_secs |
10 | Result reads failing this long make waiting callers run direct |
redis_max_connections |
32 | Raised automatically to worker slots + 16 |
Redis keys are cluster-safe (hash-tagged per request and per model+target), so cluster-mode Valkey (e.g. ElastiCache Serverless) works.
Providers
ModelConfig(provider="openai", model_id="llama-3.1-70b",
targets={"primary": Target(base_url="http://vllm:8000/v1")}, api_key_env="LOCAL_LLM_KEY")
QueueConfig(allowed_providers={"bedrock"}, denied_hosts=["*.openai.azure.com"]) # enforced on sender and worker
QueueConfig(providers={"myprov": "mypkg.providers:MyProvider"}) # custom provider, loaded by every pod
A custom provider implements async call(op, params, target) -> dict and classify(exc) -> ErrorInfo(kind="throttle" | "retryable" | "fatal", retry_after_secs). It can also be registered through the taskiq_llm_queue.providers entry point. SDK-level retries are off inside workers (botocore total_max_attempts=1, openai max_retries=0), so throttles reach the limiter.
Lanes and queues
| Lane | Queue | Use |
|---|---|---|
rt |
llm-<model>-rt |
Live user requests; polled first; may use the whole limit |
bg |
llm-<model> |
Background work; at most 80 % of the limit |
bulk |
llm-<model>-bulk if configured (e.g. embeddings), else base |
Ingestion/batch; lowest priority |
A queued bg/bulk call waits at most 15 s for a slot, then goes back to the queue with a delay instead of holding a worker.
Results
Results are stored as JSON (never pickle) for 5 minutes and read once (GETDEL). Each pod runs one result dispatcher: a pub/sub wake reads only the announced results, and a periodic batched sweep catches the rest, so 10K waiting calls don't mean 10K polling loops.
Fallbacks: how to check and update
A fallback chain is tried inside the worker. Each step is retried with backoff before moving on:
- same model, primary target
- same model, another target (e.g. a cross-region profile
gpt120b@apac) - another model (only when the call's policy is
chain)
Embeddings never switch model. Throttles shrink the limiter but never trip the circuit breaker; only 5xx and timeouts do.
uv run llm-queue fallback get # whole config (JSON); starts from your app's defaults
uv run llm-queue fallback get --model gpt120b # resolved chain for one model
uv run llm-queue fallback test --model gpt120b # dry run: which target would serve now
uv run llm-queue fallback set --model gpt120b --chain "gpt20b"
uv run llm-queue fallback set --file fallback.json # merges per model; --replace to overwrite all
uv run llm-queue fallback reset # back to the startup default
Chains are validated and reloaded by every pod within 30 s with no restart. An invalid config is rejected and the last good one stays. Every change is written to an audit stream.
{ "chains": {
"gpt120b": ["gpt20b"],
"gpt20b": [{"model": "gpt120b", "require_headroom": true}]
},
"skip_targets": { "gpt120b": ["apac"] } }
To see fallbacks happen: res.meta / response_metadata show served_model, served_target, fallback_used and attempts, and the llmq_fallback_total{model,step} metric counts them. samples/05_fallback_on_throttle.py shows it end to end.
Operating it
uv run llm-queue routing set --model gpt20b --lane rt --pct 10 # canary: 10 % of sessions via SQS
uv run llm-queue routing kill on # everything direct within 30 s
uv run llm-queue routing reset
uv run llm-queue limiter show --model gpt120b # learned limit, in-flight, pause
uv run llm-queue limiter reset --model gpt120b [--target apac] # forget the learned limit
uv run llm-queue breaker show | reset
uv run llm-queue queues # messages per queue and DLQ
uv run llm-queue job send ingest --payload-file payload.json --queue jobs-ingest
Failure behaviour
| Failure | What happens |
|---|---|
| Provider throttles | Limiter shrinks and pauses that model + target; brokers stop receiving it unless another target in the chain has headroom; work waits in SQS |
| Provider down (5xx, timeouts) | Circuit breaker opens → next fallback step |
| Worker killed mid-task | Message reappears after the visibility timeout; the claim prevents a second provider call |
| Rolling deploy | Pods stop receiving at once, release unstarted work, finish running calls, then exit (90 s budget in the reference deployment) |
| SQS unreachable | That model switches to direct (+ shared limiter) for 300 s on transient send errors only. An ambiguous send is claimed first, so it can't run twice |
| Redis/Valkey unreachable | Workers release messages instead of losing them; senders run direct with a local per-pod cap (last known limit ÷ live pods) |
Logging
The library logs with structlog's key-value API on top of stdlib logging and never configures logging itself. Records are named taskiq_llm_queue.*; the message reads event key=value ... and the fields are also on record.llmq (a dict). Prompts, responses and params are never logged.
import logging
logging.getLogger("taskiq_llm_queue").setLevel(logging.INFO) # stdlib apps: that's all
# loguru apps: the usual InterceptHandler picks these up like any stdlib logger.
# structlog apps: use structlog.stdlib.ProcessorFormatter on your handler to render them.
The llm-queue CLI logs warnings and errors to stderr, so its stdout is always plain JSON.
Security and data
- Encryption is server-side on the queues (SSE-SQS or SSE-KMS); the library needs no setting for either.
- Prompts and responses are never logged.
user_id/session_idare hashed (keyed BLAKE2b) in task labels; raw ids stay inside the message body. - Nothing secret goes into messages; credentials come from the AWS chain (IRSA / Pod Identity) or env.
- Unknown task names or newer message versions are released for another pod, never silently dropped.
Run it locally (no AWS needed)
make up # Valkey (password) + ElasticMQ + fakellm
make test # unit + integration against the stack
make down
# Windows without make: uv run python tools/llmq_dev.py up | test | down
Runnable examples are in samples/: quickstart, chat, embeddings, lanes, fallback on throttle, send-only and worker pods, LangChain, LangGraph, jobs, cancel and timeouts, sync calls, CLI, and 13_research_assistant_flow.py, which replays a real app's flows locally or against real Bedrock (--target aws, settings from .env or AWS Secrets Manager). The stack, capability checks and test layers are described in docs/TESTING.md; delivery-safety rules in docs/DESIGN.md.
Credits
Broker design and feature set build on taskiq-aio-sqs (MIT, © 2022-2024 Jesse Constante) and taskiq-sqs (MIT, © 2026 Mike Shultz and the Taskiq team). Licence: MIT.
Metadata
Release files for taskiq-llm-queue 0.1.0a1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| taskiq_llm_queue-0.1.0a1.tar.gz | 165.5 kB | With provenance | Details for taskiq_llm_queue-0.1.0a1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| taskiq_llm_queue-0.1.0a1-py3-none-any.whl | Python 3 | none | any | Details for taskiq_llm_queue-0.1.0a1-py3-none-any.whl |
Total release size: 345.4 kB
Release files / taskiq_llm_queue-0.1.0a1.tar.gz
| Download URL | taskiq_llm_queue-0.1.0a1.tar.gz |
|---|---|
| Size | 165.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
fa7942009dca099f40daa9e472dc3f90c7e9edddc49a4928448397a7513bd6ab
|
|
BLAKE2b-256 checksum How to use checksums |
6dbcdb3dc2fccc77aa04ddbea165a42ffaa37f2f6a8482a011202ad349df0845
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 29, 2026.
Transparency logRelease files / taskiq_llm_queue-0.1.0a1-py3-none-any.whl
| Download URL | taskiq_llm_queue-0.1.0a1-py3-none-any.whl |
|---|---|
| Size | 179.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
865e08eb76fba5a7482464e93a488f7fd70bd845314d2da8a8af7bdfc1f1a492
|
|
BLAKE2b-256 checksum How to use checksums |
d0f489281104fb38fd5250a7af10df27eed8225edd9d01e790e94ff2ee93a3f4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 29, 2026.
Transparency logSecurity for taskiq-llm-queue 0.1.0a1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for taskiq_llm_queue-0.1.0a1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 29, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.