taskiq-llm-queue
A durable, throttle-aware queue for LLM calls: LangChain-compatible, self-tuning, no extra servers.
Architecture · Samples · CLI · Changelog
Status: alpha (
0.2.1). Built and tested: theLLMQueueclient, the queue broker on Amazon SQS or Valkey Streams, adaptive limiter, claims, circuit breaker, retry and fallback, JSON and S3 result backends, Bedrock and OpenAI-compatible providers, LangChain/LangGraph adapters, canary routing, failover and the CLI. 869 tests (unit, integration and the scenario harness) on Valkey 8.1, ElasticMQ 1.7.1 and a fake LLM server; CI on Python 3.11–3.14. Breaking changes are still possible before 1.0 (see the CHANGELOG).
Contents
| Start | Overview · Highlights · Install · Quickstart |
| Use | Client API · LangChain and LangGraph · Providers |
| Configure | Configuration · Settings per provider and model · Retry backoff · Setting names |
| Queues | Lanes and queues · Queue backend · Concurrency · Rate limits |
| Run | Results · Fallbacks · Operating it · Failure behaviour · Logging · Security |
| Develop | Run it locally · Samples · Further reading · Credits |
Overview
An LLM request client with its own durable queue, built on taskiq and Amazon SQS (or Valkey Streams).
Your code keeps calling LangChain models (ainvoke, aembed_query) or a small client API. Each call becomes a task on a queue. Workers in your own app's processes run it under a limit learned from the provider's throttling, retry and fall back when needed, and return the result through Redis/Valkey.
your code ──> client / LangChain adapter ──> SQS or Valkey Streams (per model: -rt queue first, base, bulk)
│
result <── Redis/Valkey <── worker in your pods ──> Bedrock / OpenAI-compatible
(per-pod dispatcher) (adaptive limiter, retry, fallback)
How it works in depth: docs/ARCHITECTURE.md walks through the components, every data flow (happy path, fallback, failover, jobs, cancellation, crash recovery), the adaptive limiter, the Valkey key schema and the full configuration reference.
Highlights
| Feature | What you get | |
|---|---|---|
| 🧺 | Spikes wait in a queue | Throttled calls aren't lost or retried in a storm |
| 🎛️ | No quota numbers to maintain | The limit for each model and target is learned from throttle errors and shared by all pods |
| ⚡ | Chat first | Separate -rt queues and reserved capacity keep live traffic ahead of background and bulk work |
| 🧩 | No extra servers | The same app pods send and execute; SQS (or Valkey) is only the broker |
| 🔌 | Drop-in | LangChain/LangGraph code doesn't change: tool calling, callbacks (Langfuse, LangSmith) and response objects are identical |
| 🛟 | Safe rollout | direct mode uses the same limiter, and a live canary moves traffic per model and lane without restarts |
| 🔁 | Fallback chains | Another target, then another model, retried with backoff inside the worker |
| 📏 | Caps and rate limits | Per model, lane and provider concurrency caps, RPM / TPM budgets, all changeable live |
Install
uv add "taskiq-llm-queue[langchain-aws]" # Bedrock + LangChain
uv add "taskiq-llm-queue[openai,langchain-openai]" # OpenAI-compatible + LangChain
uv add "taskiq-llm-queue[speedups]" # hiredis parser for redis-py
| Extra | Packages |
|---|---|
| (core) | taskiq, aiobotocore, redis>=5.0.1, pydantic>=2.10, orjson, structlog |
openai |
openai, httpx |
langchain-aws / langchain-openai |
LangChain 0.3 and 1.x are both supported |
metrics |
prometheus-client |
speedups |
hiredis |
The package declares version ranges; pin exact versions in your app's lockfile. aiobotocore pins a narrow botocore window, so let your resolver pick an aiobotocore that matches your boto3.
Quickstart
from taskiq_llm_queue import LLMQueue, ModelConfig, QueueConfig, Target, request_context
from taskiq_llm_queue.langchain import QueuedChatBedrockConverse
llmq = LLMQueue(QueueConfig(
key_prefix="llmq:local",
redis_host="localhost", redis_port=6379, redis_password="localdev", redis_ssl=False,
sqs_endpoint_url="http://localhost:9324", # ElasticMQ locally; omit on AWS
models={
"gpt120b": ModelConfig(
provider="bedrock", model_id="openai.gpt-oss-120b-1:0",
targets={"primary": Target(region="ap-south-1"),
"apac": Target(profile_id="apac.openai.gpt-oss-120b-1:0")},
queues={"rt": "llm-gpt120b-rt", "default": "llm-gpt120b"}),
},
)) # or QueueConfig.from_env() (LLMQ_* variables)
async def main():
await llmq.start(mode="both") # "send" | "work" | "both"
model = QueuedChatBedrockConverse(llmq=llmq, model_alias="gpt120b")
with request_context(lane="rt", session_id="s-1", user_id="u-1"):
reply = await model.ainvoke("Summarise today's market in one line.")
print(reply.content, reply.response_metadata["served_model"])
await llmq.stop() # drains in-flight work
Tip: in a FastAPI app, call
start()in the lifespan startup andstop()on shutdown. Send-only processes, such as CLIs and scripts, usemode="send".
Client API
Every call returns a result object with .meta: served_model, served_target, fallback_used, attempts, queue_wait_ms, provider_ms, worker.
res = await llmq.converse("gpt120b", {"messages": [...], "system": [...], "toolConfig": ...},
lane="rt", fallback="chain", timeout_secs=90, request_id="req-1")
res.response # exact Bedrock Converse dict
emb = await llmq.embed("titan", ["a", "b"], lane="bulk")
emb.vectors # list[list[float]]; `emb` itself also works as a list
chat = await llmq.chat("llama", {"messages": [{"role": "user", "content": "hi"}]})
chat.response # OpenAI-compatible chat completion dict
raw = await llmq.invoke_model("titan", {"inputText": "a"}, content_type="application/json")
raw.json(), raw.body, raw.content_type
await llmq.cancel("req-1") # not run if queued, stopped if running; caller gets LLMQueueCancelled
| Topic | Rule |
|---|---|
| Timing | timeout_secs (relative) or deadline_at (absolute epoch seconds) on every call |
| Model ids | Aliases, optionally with a target: gpt120b, gpt120b@apac. Unknown aliases are rejected when the config is built |
| Errors | Everything derives from LLMQueueError: LLMQueueTimeout, LLMQueueCancelled, ProviderError(retryable, retry_after_secs), QueueUnavailable, ConfigError |
| Sync callers | converse_sync, chat_sync, embed_sync, invoke_model_sync run direct under the shared limiter (counted as sync fallbacks). Set strict_async=True to raise instead. close_sync() (or stop()) closes the sync runner |
| Jobs | register_job(name, fn) on workers, send_job(name, payload, timeout_secs=...) anywhere; jobs run on the job queues with the same claims and DLQ rules |
LangChain and LangGraph
| Class | Based on | How |
|---|---|---|
QueuedChatBedrockConverse |
langchain_aws.ChatBedrockConverse |
Builds the request and parses the reply with the installed langchain-aws's own code, so bind_tools, structured output, cache usage and service tier match exactly |
QueuedBedrockEmbeddings |
langchain_aws.BedrockEmbeddings |
aembed_query / aembed_documents go through the queue |
QueuedChatOpenAI |
langchain_openai.ChatOpenAI |
http_async_client with a queued httpx transport: chat completions and embeddings go through the queue; queue meta lands in response_metadata. The Responses API (/responses) is not supported yet |
QueuedOpenAIEmbeddings |
langchain_openai.OpenAIEmbeddings |
Same transport; check_embedding_ctx_length=False by default for non-OpenAI servers |
- Model names: pass the queue alias as
model_alias=(plusmodel_id=orllmq=). Passing an alias asmodel=raises a clear error. - Streaming: all queued classes force
disable_streaming=True/streaming=False.astream()and LangGraphstream_mode="messages"return one complete message, so nothing bypasses the queue. - Laziness: queued classes find the running
LLMQueueon first use and check routing and the kill switch on every call, so you can create them at import time. - Tracing: Langfuse / LangSmith see the same run name (the parent class name) and model as a direct call; after a fallback, the served model id is reported.
Providers
ModelConfig(provider="openai", model_id="llama-3.1-70b",
targets={"primary": Target(base_url="http://vllm:8000/v1")}, api_key_env="LOCAL_LLM_KEY")
QueueConfig(allowed_providers={"bedrock"}, denied_hosts=["*.openai.azure.com"]) # enforced on sender and worker
QueueConfig(providers={"myprov": "mypkg.providers:MyProvider"}) # custom provider, loaded by every pod
A custom provider implements async call(op, params, target) -> dict and classify(exc) -> ErrorInfo(kind="throttle" | "retryable" | "fatal", retry_after_secs). It can also be registered through the taskiq_llm_queue.providers entry point. SDK-level retries are off inside workers (botocore total_max_attempts=1, openai max_retries=0), so throttles reach the limiter.
Configuration
QueueConfig.from_env() reads LLMQ_* variables (durations end in _SECS; older _S spellings are not read). Notable settings:
Connections and backends
| Setting | Default | Note |
|---|---|---|
aws_region |
None |
Left to the AWS credential chain / IRSA |
queue_backend (LLMQ_QUEUE_BACKEND) |
sqs |
valkey puts every queue on Valkey Streams; see Queue backend |
queue_redis_url (LLMQ_QUEUE_REDIS_URL) |
None |
With valkey: a separate Valkey for the queues (else the state Valkey) |
redis_url or redis_host/port/username/password/ssl |
localhost | redis_password is a SecretStr |
redis_socket_timeout_secs / connect_timeout / health_check_interval |
5 / 2 / 30 | Connection errors retried 3× with backoff |
redis_max_connections |
32 | Raised automatically to worker slots + 16 |
id_hash_salt (LLMQ_ID_HASH_SALT) |
library constant | Set your own in production; keys the BLAKE2b hash of session/user ids |
s3_offload_bucket (LLMQ_S3_OFFLOAD_BUCKET) |
none | Messages over inline_max_bytes (200 KB) carry their params in S3 under s3_offload_prefix instead of Valkey; add a lifecycle rule to expire leftovers |
Timeouts, retries and failover
| Setting | Default | Note |
|---|---|---|
timeout_secs per lane |
rt 90, bg 300, bulk 900 | |
attempt_timeout_secs |
180 | One provider attempt; a hung target moves to the next fallback step |
provider_read_timeout_secs / provider_connect_timeout_secs |
300 / 10 | Bedrock botocore and OpenAI httpx timeouts |
throttle_retry_attempts |
6 | Tries per fallback step for throttles (backoff holds no slot) |
retry_base_secs / retry_multiplier / retry_spread / retry_max_secs |
1 / 4 / 2.5 / 30 | Retry n sleeps a random time in [base·4ⁿ, base·4ⁿ·2.5]: 1–2.5 s, 4–10 s, 16–30 s, then 30 s; see Retry backoff |
throttle_retry_max_secs |
10 | Cap of one sleep after a throttle, so 6 throttle tries fit an rt deadline |
rt_capacity_wait_secs |
30 | How long a received rt call keeps looking for a free slot before no_capacity (never past its deadline) |
sqs_send_timeout_secs |
5 | SQS counts as down only when no send completed for this long; then the call runs direct |
sqs_timeouts_to_failover |
3 | Send timeouts within 60 s before the model's later calls skip SQS for sqs_failover_secs (a refused send: at once) |
sqs_batch_wait_secs |
0.02 | bg and bulk sends within this window share one SendMessageBatch (0 = off); rt never batches |
valkey_wait_failover_secs |
10 | Result reads failing this long make waiting callers run direct |
Capacity and limiter
| Setting | Default | Note |
|---|---|---|
worker_slots (LLMQ_WORKER_SLOTS=gpt120b:50,jobs:4) |
50 per model alias (4 for jobs) |
Tasks one pod runs at once for that alias's queues |
ModelConfig.limiter_initial (LLMQ_AC_INITIAL='{"gpt120b": 40}') |
20 | Start of the adaptive limit before anything was learned |
ModelConfig.max_concurrency (LLMQ_AC_MAX_BY_MODEL='{"gpt120b": 30}') |
None (adaptive up to LLMQ_AC_MAX) |
Hard cap on calls in flight across all pods, per alias and target; see Concurrency |
limiter.max_limit (LLMQ_AC_MAX) |
2000 | Global ceiling of every adaptive limit |
limiter.bg_share (LLMQ_AC_BG_SHARE) |
0.8 | Fraction of a limit bg/bulk may use; rt may use all of it |
settings_default (LLMQ_SETTINGS, JSON) |
{} |
Per-provider and per-model values of the settings below; see Settings per provider and model |
Redis keys are cluster-safe (hash-tagged per request and per model+target), so cluster-mode Valkey (e.g. ElastiCache Serverless) works.
Settings per provider and model
A provider (bedrock, openai, or a custom one) serves several models. These settings can be given once for a provider, overridden for one model, and changed live on every pod without a restart:
max_concurrency, limiter_initial, bg_share, timeout_secs.<lane>, embed_timeout_secs.<lane>, attempt_timeout_secs, throttle_retry_attempts, retry_base_secs, retry_multiplier, retry_spread, retry_max_secs, throttle_retry_max_secs, breaker_threshold, breaker_cooldown_secs, lane_max_concurrency.<lane>, the rate budget rpm, tpm, rate_window, rate_burst, rt_rate_share (Rate limits), and for a provider only shared_max_concurrency, shared_rpm, shared_tpm.
Each one resolves through six layers; the highest that has a value wins:
| # | Layer | Where it comes from |
|---|---|---|
| 6 | Live model | Valkey, llm-queue settings set --model M ... |
| 5 | Live provider | Valkey, llm-queue settings set --provider P ... |
| 4 | Env model | LLMQ_SETTINGS models, plus ModelConfig.max_concurrency / limiter_initial and LLMQ_AC_MAX_BY_MODEL / LLMQ_AC_INITIAL |
| 3 | Env provider | LLMQ_SETTINGS providers |
| 2 | Env global | LLMQ_ATTEMPT_TIMEOUT_SECS, LLMQ_TIMEOUT_RT_SECS, LLMQ_RETRY_*, LLMQ_CB_*, LLMQ_AC_BG_SHARE, ... (code, .env or Secrets Manager) |
| 1 | Built-in default | The defaults in the tables above |
# .env: provider defaults and one model override (layers 3-4), validated at boot
LLMQ_SETTINGS='{"providers": {"openai": {"shared_max_concurrency": 40, "attempt_timeout_secs": 120}},
"models": {"gpt120b": {"max_concurrency": 30, "timeout_secs": {"rt": 60}}}}'
# live, reaches every pod within LLMQ_CFG_RELOAD_SECS (30 s); every change is audited
llm-queue settings set --model gpt120b max_concurrency=20 timeout_secs.rt=45
llm-queue settings set --provider bedrock attempt_timeout_secs=90 retry_multiplier=3
llm-queue settings get --model gpt120b # every setting with its value and source
llm-queue settings unset --model gpt120b max_concurrency
llm-queue settings reset # drop all live overrides
settings get shows each value's source (live:model, live:provider, env:model, env:provider, env, default), so a value that differs from your .env is easy to explain. A live value wins over .env until it is unset. An unknown alias or provider, an unknown setting or a bad value is refused before anything is written. If the stored value is unreadable or Valkey is down, pods keep the last good one.
| Topic | Detail |
|---|---|
| Provider-wide cap | shared_max_concurrency on a provider bounds the calls in flight across all its models and pods, e.g. one vLLM server hosting three models. Each call then holds two slots, its model's and the provider's. Measured (S55): shared cap 8 on bedrock, gpt20b and gpt120b allowed 20 each, 600 calls peaked at exactly 8, all OK, none duplicated |
| Live bucket change | Measured (S54): gpt20b at max_concurrency 20 on 3 workers; settings set --model gpt20b max_concurrency=5 mid-run brought the peak from 20 to 5 after one reload, 900/900 OK |
| Single-target models | No fallback target means a call fails after throttle_retry_attempts (6) throttles in a row: 0.3⁶ ≈ 0.07 % of calls under a 30 % throttle storm (S35b: 4 of 3,020). Add a fallback, or raise throttle_retry_attempts (10 tries: 0.3¹⁰ ≈ 0.0006 %) |
| Errors carry meta | A failed call raises ProviderError / LLMQueueTimeout with .meta (attempts, queue wait, last target tried) |
| Not live | Needs a restart: connection settings, the model list (targets, queues, provider), credentials, provider HTTP timeouts, worker_slots and retry_attempts |
Retry backoff
Each fallback step retries a failed attempt retry_attempts times (3), or throttle_retry_attempts times (6) once the provider throttled it. Retry n (0-based) sleeps a random time in [base · multiplierⁿ, base · multiplierⁿ · spread], capped at retry_max_secs:
| Retry | Defaults (base 1, multiplier 4, spread 2.5, cap 30) | After a throttle (cap 10) |
|---|---|---|
| 1st | 1–2.5 s | 1–2.5 s |
| 2nd | 4–10 s | 4–10 s |
| 3rd | 16–30 s | 10 s |
| 4th and later | 30 s | 10 s |
A provider Retry-After larger than the drawn delay wins, up to retry_max_secs. A sleep that would pass the call's deadline is skipped and the next fallback step is tried. No limiter slot is held while sleeping. All five knobs are LLMQ_* variables and can be set per provider or model like the settings above.
A sleeping call frees its limiter slot but keeps its worker task (one of worker_slots). Under a long throttle storm a longer curve therefore lowers per-pod throughput; if queued callers start to time out, raise worker_slots or lower retry_multiplier / throttle_retry_max_secs for that provider or model. The scenario harness scales the whole curve by 0.2 (base 0.2 s, caps 6 s and 2 s) to keep runs short.
Setting names
Every duration ends in _SECS (milliseconds in _MS). Old spellings from pre-release builds are not read: a variable with one of them is silently ignored and the default applies. Rename them in your .env, Helm values and Secrets Manager entries.
Old spellings and their replacements
| Old spelling (ignored) | Use |
|---|---|
LLMQ_CFG_RELOAD_S |
LLMQ_CFG_RELOAD_SECS |
LLMQ_CONSUMER_DRAIN_S |
LLMQ_CONSUMER_DRAIN_SECS |
LLMQ_RETRY_BASE_S |
LLMQ_RETRY_BASE_SECS |
LLMQ_SQS_WAIT_TIME_S |
LLMQ_SQS_WAIT_TIME_SECS |
LLMQ_SQS_FAILOVER_S |
LLMQ_SQS_FAILOVER_SECS |
LLMQ_STATE_TTL_S |
LLMQ_STATE_TTL_SECS |
LLMQ_AC_PAUSE_S |
LLMQ_AC_PAUSE_SECS |
LLMQ_AC_LEASE_TTL_S |
LLMQ_AC_LEASE_TTL_SECS |
LLMQ_CB_WINDOW_S |
LLMQ_CB_WINDOW_SECS |
LLMQ_CB_COOLDOWN_S |
LLMQ_CB_COOLDOWN_SECS |
LLMQ_TIMEOUT_RT / _BG / _BULK |
LLMQ_TIMEOUT_RT_SECS / _BG_SECS / _BULK_SECS |
LLMQ_TIMEOUT_EMBED_RT / _BG / _BULK |
LLMQ_TIMEOUT_EMBED_RT_SECS / _BG_SECS / _BULK_SECS |
Lanes and queues
| Lane | Queue | Use |
|---|---|---|
rt |
llm-<model>-rt |
Live user requests; polled first; may use the whole limit |
bg |
llm-<model> |
Background work; at most 80 % of the limit |
bulk |
llm-<model>-bulk if configured (e.g. embeddings), else base |
Ingestion/batch; lowest priority |
A queued bg/bulk call waits at most 15 s for a slot, then goes back to the queue with a delay instead of holding a worker.
Queue backend: SQS or Valkey Streams
Queues live on Amazon SQS by default. LLMQ_QUEUE_BACKEND=valkey puts them on Valkey Streams instead, with the same queue names and the same delivery rules: visibility timeout with heartbeat, release with a delay, redelivery after a pod dies, a DLQ after maxReceiveCount, delays, expiry, strict lane priority and batched send / ack. The broker is unchanged: ValkeyQueueClient answers the SQS calls it makes (docs/VALKEY_QUEUE_PLAN.md).
| SQS | Valkey Streams | |
|---|---|---|
| Create queues | your infrastructure code | llm-queue queues create [--visibility-secs 60 --max-receives 10 --retention-secs 345600] |
| Where | AWS, multi-AZ, managed | LLMQ_QUEUE_REDIS_URL, or the state Valkey when unset |
| Valkey down | calls run direct; queued work stays in SQS | queue, limiter and results are down together; calls run direct |
| Backlog | cheap, days | Valkey memory: ~600 B per message on top of the body (bodies ≤ 200 KB; larger params by reference) |
| Metrics | CloudWatch | llmq_queue_depth{queue,kind}, llmq_queue_oldest_seconds{queue} from the pods |
| DLQ redrive | console / StartMessageMoveTask |
llm-queue queues redrive --queue <name> |
| FIFO queues | yes | no (refused at boot) |
Production Valkey for queues: a dedicated server with appendonly yes, appendfsync everysec, maxmemory-policy noeviction, and a replica (Sentinel or Multi-AZ). A failover can lose or repeat the last messages (handlers are idempotent; a loss shows as a caller timeout). Measured on a laptop: everysec 1,513 receive+delete/s, always 578/s, 0 messages lost after a hard kill in either mode.
Concurrency: queue depth vs bucket size
The queue depth is unbounded: the queue holds every request that is not running. How many run at once is set by three knobs:
| Knob | Scope | Behaviour |
|---|---|---|
worker_slots |
per pod, per model alias (default 50) | Tasks one pod runs at once. Total pod capacity is pods × worker_slots |
| The limiter | cluster-wide, per model alias and target, in Valkey | Provider calls in flight across all pods. Starts at limiter_initial (20), grows on busy successes, is cut on clustered throttles, never exceeds LLMQ_AC_MAX (2000) |
max_concurrency |
cluster-wide, per model alias and target (default unset) | Hard cap on that limit. With max_concurrency=10, at most 10 calls of that alias run against each of its targets at once, whatever the pod count |
- A model with two targets can have 2 ×
max_concurrencyin flight. The limiter still adapts below the cap. - Set
limiter_initial == max_concurrencyfor a fixed bucket that only shrinks under real throttling; alimiter_initialabove the cap is clamped to it (logged once aslimiter_initial_clamped). Lowering the cap also clamps a limit learned earlier, on its next use. bgandbulkmay usebg_share(80 %) of the limit, so with a cap of 10, background work runs at most 8 at once and keeps 2 slots forrt. SetLLMQ_AC_BG_SHARE=1to give all lanes the whole bucket.
A bucket per lane. lane_max_concurrency.<lane> gives one lane of a model its own cap inside the model's bucket, per target and across all pods, e.g. bulk embeddings at most 5 while bg may use 15 of 20:
LLMQ_SETTINGS='{"models": {"titan": {"max_concurrency": 20, "lane_max_concurrency": {"bulk": 5, "bg": 15}}}}'
llm-queue settings set --model titan lane_max_concurrency.bulk=3 # live, every pod
A call in a capped lane holds a slot of the model's bucket and one of the lane's ({m:<alias>#<lane>:<target>}, a fixed bucket that never learns). A full lane leaves its own backlog in the queue while the other lanes keep receiving. Measured (S57): gpt20b bucket 20, bulk 3, bg 5, 300 bulk + 300 bg calls at once: bulk peaked at exactly 3, bg at exactly 5, an rt call in the middle started in 0.23 s, all 600 OK.
How receiving respects the bucket
A pod receives at most min(free worker slots, limiter headroom − its own calls already waiting for a slot), so a full bucket leaves the backlog in the queue rather than in pod memory. The rest wait until their lane deadline (timeout_secs: rt 90 s, bg 300 s, bulk 900 s). bg/bulk calls that do get received but find no slot within 15 s go back to the queue with a delay. An rt call that is received keeps walking its fallback chain for a free slot (2 s per step) for up to rt_capacity_wait_secs (30 s, never past its deadline), then fails with no_capacity.
Size the bucket so that cap × calls per second per slot × deadline covers the backlog you expect. max_concurrency and bg_share can also be set per provider and changed live (Settings per provider and model), and a provider can have one bucket shared by all its models. While Valkey is down, each pod runs under a local cap of last limit / live pods, never above max_concurrency. This is best effort: with the pod count unknown, each pod may run up to the cap.
Measured (S53, fakellm at 200 ms p50, 3 workers × 50 slots, max_concurrency=10, bg_share=1): 2,000 bg calls queued (2,000 visible in SQS at peak), peak provider concurrency exactly 10 (from the call log and fakellm's own counter), limiter never above 10, 2,000/2,000 done, 0 lost, 0 duplicates, 61 s wall time. SQS in-flight peaked at 40: 10 running plus finished messages waiting for the 0.5 s batched delete.
Rate limits (RPM / TPM)
Concurrency caps how many calls run at once; a provider quota is usually per minute. Set rpm (calls that may start per minute) and / or tpm (tokens per minute) per model, and shared_rpm / shared_tpm per provider for an account quota. Each budget is per model target (a model with a primary and an APAC profile has one budget on each), shared by every pod in Valkey. Unset means no rate limit.
LLMQ_SETTINGS='{"models": {"gpt120b": {"rpm": 500, "tpm": 200000}},
"providers": {"bedrock": {"shared_rpm": 800}}}'
llm-queue settings set --model gpt120b rpm=300 rate_window=fixed # live, every pod
llm-queue status --model gpt120b # waiting / working / started this minute / left
A call starts only with a rate token and a concurrency slot.
| Setting | Default | Meaning |
|---|---|---|
rate_window |
smooth |
smooth: a token bucket, rpm / 60 per second with rate_burst back to back. fixed: the whole rpm from each calendar minute (500 at 10:00:00, the next 500 from 10:01:00) |
rate_burst |
10 % of rpm |
Calls that may start back to back in smooth mode; TPM uses the same fraction |
rt_rate_share |
0.2 | Part of each budget only rt may use: bg/bulk get at most 80 % of the rate, so a backlog never starves live calls |
When a budget is used up:
| Who | What happens |
|---|---|
bg / bulk |
Wait for their own target's budget: the message goes back to the queue with a delay until the next token, and that pod stops receiving the model's bg/bulk work until then. They never spend a fallback target's budget |
rt |
Moves to the next target in its chain (each target has its own budget), keeps trying for rt_capacity_wait_secs (30 s, never past its deadline), then fails with ProviderError("rate_limited ..."), retryable=True, retry_after_secs = time to the next token (HTTP 429 through http_transport) |
| TPM | Reserves input characters / 4 + max output tokens (1,000 when the request sets none) before the call; corrected with the response's usage after it, so a generous max_tokens does not hold the budget |
| Valkey down | Each pod applies limit / live pods in process (best effort; the total is right while the pod count is right) |
llm-queue status prints per model: waiting / delayed / received per lane queue, and per target working (calls in flight, all pods), limit, started this minute, rpm_left, tokens, tpm_left, plus the provider's shared budget. Metrics: llmq_rate_used and llmq_rate_remaining per model target. The adaptive limiter stays on underneath: if the real quota is lower than the rpm you set, throttles still cut concurrency.
Results
Results are stored as JSON (never pickle) for 5 minutes and read once (GETDEL). Each pod runs one result dispatcher: a pub/sub wake reads only the announced results, and a periodic batched sweep catches the rest, so 10K waiting calls don't mean 10K polling loops.
Fallbacks: how to check and update
A fallback chain is tried inside the worker. Each step is retried with backoff before moving on:
- same model, primary target
- same model, another target (e.g. a cross-region profile
gpt120b@apac) - another model (only when the call's policy is
chain)
Embeddings never switch model. Throttles shrink the limiter but never trip the circuit breaker; only 5xx and timeouts do.
uv run llm-queue fallback get # whole config (JSON); starts from your app's defaults
uv run llm-queue fallback get --model gpt120b # resolved chain for one model
uv run llm-queue fallback test --model gpt120b # dry run: which target would serve now
uv run llm-queue fallback set --model gpt120b --chain "gpt20b"
uv run llm-queue fallback set --file fallback.json # merges per model; --replace to overwrite all
uv run llm-queue fallback reset # back to the startup default
Chains are validated and reloaded by every pod within 30 s with no restart. An invalid config is rejected and the last good one stays. Every change is written to an audit stream.
{ "chains": {
"gpt120b": ["gpt20b"],
"gpt20b": [{"model": "gpt120b", "require_headroom": true}]
},
"skip_targets": { "gpt120b": ["apac"] } }
To see fallbacks happen: res.meta / response_metadata show served_model, served_target, fallback_used and attempts, and the llmq_fallback_total{model,step} metric counts them. samples/05_fallback_on_throttle.py shows it end to end.
Operating it
uv run llm-queue routing set --model gpt20b --lane rt --pct 10 # canary: 10 % of sessions via the queue
uv run llm-queue routing kill on # everything direct within 30 s
uv run llm-queue routing reset
uv run llm-queue limiter show --model gpt120b # learned limit, in-flight, pause
uv run llm-queue limiter reset --model gpt120b [--target apac] # forget the learned limit
uv run llm-queue breaker show | reset
uv run llm-queue settings get --model gpt120b # effective settings + source
uv run llm-queue settings set --model gpt120b max_concurrency=20 # live bucket size, all pods
uv run llm-queue settings set --provider openai shared_max_concurrency=40
uv run llm-queue queues # messages per queue and DLQ
uv run llm-queue job send ingest --payload-file payload.json --queue jobs-ingest
The full command reference is in samples/12_cli.md.
Failure behaviour
| Failure | What happens |
|---|---|
| Provider throttles | Limiter shrinks and pauses that model + target; brokers stop receiving it unless another target in the chain has headroom; work waits in the queue |
| Provider down (5xx, timeouts) | Circuit breaker opens → next fallback step |
| Worker killed mid-task | Message reappears after the visibility timeout; the claim prevents a second provider call |
| Rolling deploy | Pods stop receiving at once, release unstarted work, finish running calls, then exit (90 s budget in the reference deployment) |
| SQS unreachable | That model switches to direct (+ shared limiter) for 300 s on transient send errors only. An ambiguous send is claimed first, so it can't run twice |
| Redis/Valkey unreachable | Workers release messages instead of losing them; senders run direct with a local per-pod cap (last known limit ÷ live pods). With the Valkey queue backend, queue failover ends as soon as Valkey is back |
Logging
The library logs with structlog's key-value API on top of stdlib logging and never configures logging itself. Records are named taskiq_llm_queue.*; the message reads event key=value ... and the fields are also on record.llmq (a dict). Prompts, responses and params are never logged.
import logging
logging.getLogger("taskiq_llm_queue").setLevel(logging.INFO) # stdlib apps: that's all
# loguru apps: the usual InterceptHandler picks these up like any stdlib logger.
# structlog apps: use structlog.stdlib.ProcessorFormatter on your handler to render them.
To trace one call on a worker, set taskiq_llm_queue.core to DEBUG. Each call then logs call_started (with queue_wait_ms), slot_acquired / slot_unavailable (waited_ms), attempt_done (target, outcome, provider_ms), backoff (sleep_secs), rt_capacity_wait, call_requeued and call_done (ok or error code, attempts, total_ms), all with the request_id. Below DEBUG these cost one level check per call.
The llm-queue CLI logs warnings and errors to stderr, so its stdout is always plain JSON.
Security and data
- Encryption is server-side on the queues (SSE-SQS or SSE-KMS); the library needs no setting for either.
- Prompts and responses are never logged.
user_id/session_idare hashed (keyed BLAKE2b) in task labels; raw ids stay inside the message body. - Nothing secret goes into messages; credentials come from the AWS chain (IRSA / Pod Identity) or env.
- Unknown task names or newer message versions are released for another pod, never silently dropped.
- Releases are published from GitHub Actions with PyPI trusted publishing, so every file carries a provenance attestation.
Run it locally (no AWS needed)
make up # Valkey (password) + ElasticMQ + fakellm
make test # unit + integration against the stack
make down
# Windows without make: uv run python tools/llmq_dev.py up | test | down
Samples
Runnable examples live in samples/. Every sample reads LLMQ_* overrides, so LLMQ_QUEUE_BACKEND=valkey runs all of them on Valkey Streams (see the "Settings" section of samples/README.md).
| Topic | Samples |
|---|---|
| Getting started | quickstart, chat, embeddings, lanes |
| Reliability | fallback on throttle, cancel and timeouts, retries and errors (15) |
| Deployment shapes | send-only and worker pods, sync calls, jobs |
| LangChain / LangGraph | LangChain, LangGraph, app-style LangChain + LangGraph graph (17) |
| Operations | CLI (12_cli.md), settings and buckets (14), rate limits and lane caps (16) |
| Real app flows | 13_research_assistant_flow.py: locally or against real Bedrock with --target aws, settings from .env or AWS Secrets Manager |
Further reading
| Document | Covers |
|---|---|
| ARCHITECTURE.md | Components, data flows, limiter, Valkey key schema, full configuration reference |
| DESIGN.md | Delivery-safety rules |
| TESTING.md | Local stack, capability checks, test layers, scenario harness |
| VALKEY_QUEUE_PLAN.md | Valkey Streams backend: design, server settings, measured numbers |
| CHANGELOG.md | Release history |
Credits
Broker design and feature set build on taskiq-aio-sqs (MIT, © 2022-2024 Jesse Constante) and taskiq-sqs (MIT, © 2026 Mike Shultz and the Taskiq team).
Licence: MIT.
Metadata
Release files for taskiq-llm-queue 0.2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| taskiq_llm_queue-0.2.1.tar.gz | 269.8 kB | With provenance | Details for taskiq_llm_queue-0.2.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| taskiq_llm_queue-0.2.1-py3-none-any.whl | Python 3 | none | any | Details for taskiq_llm_queue-0.2.1-py3-none-any.whl |
Total release size: 492.7 kB
Release files / taskiq_llm_queue-0.2.1.tar.gz
| Download URL | taskiq_llm_queue-0.2.1.tar.gz |
|---|---|
| Size | 269.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
97ef5bdfdb998241fa622b22a044363bfea922cf4e7796a440959dbd97dd7e24
|
|
BLAKE2b-256 checksum How to use checksums |
a574f6334fe4517a0ec3489e74fde433a748bc3408a84249934646b95728fb04
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 2, 2026.
Transparency logRelease files / taskiq_llm_queue-0.2.1-py3-none-any.whl
| Download URL | taskiq_llm_queue-0.2.1-py3-none-any.whl |
|---|---|
| Size | 222.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2ef2ef720680aaa8bba9e3dfe7311b3ecb40892f12b2fec8011a214e256d182d
|
|
BLAKE2b-256 checksum How to use checksums |
5a05585dd2f920e9239036943426206e4f4a5843a580f7754090b30636c504b7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 2, 2026.
Transparency logSecurity for taskiq-llm-queue 0.2.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for taskiq_llm_queue-0.2.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 2, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.