celery-sqs-native-delay ⏳️

celery[sqs] fetches every ETA/countdown task off the queue immediately and holds it, unacked, in the worker's own memory until the ETA arrives. SQS already has a native mechanism for this. This package makes Celery use it by wrapping kombu's existing SQS transport.

How it routes

flowchart LR
    A["task published<br/>with a delay"] --> B{".fifo queue?"}
    B -->|yes| C["celery's own<br/>ETA timer"]
    B -->|no| D{"delay ≤ 900s?"}
    D -->|yes| E["SQS native<br/>DelaySeconds"]
    D -->|no| F["EventBridge Scheduler<br/>one-time schedule"]

Install and configure

pip install celery-sqs-native-delay

app = Celery(
    "myapp",
    broker="sqs://",
    broker_transport="celery_sqs_native_delay:Transport",
    broker_transport_options={
        "eventbridge_role_arn": "arn:aws:iam::123456789012:role/celery-scheduler",
    },
)

Requirements and limitations

• Celery protocol 2 is required. Protocol 2 has been celery's default since 4.0 (2016), so almost every app already meets this. Protocol 2 puts eta in the message headers. Protocol 1 puts it in the body, which this package never inspects, so a protocol 1 delay is invisible from here and falls back silently to celery's own local timer.
• FIFO queues get no native delay. SQS has no per-message DelaySeconds on a .fifo queue, and EventBridge Scheduler can't be used either, since separate schedule invocations have no ordering guarantee relative to each other. A delayed publish to a .fifo queue falls back entirely to celery's own local ETA timer, exactly as if this package weren't installed.
  • The way to get delay on a FIFO queue without any of that is a queue-level DelaySeconds attribute set at queue creation. It applies uniformly to every message and preserves ordering.

IAM

Three things need permissions: the role in eventbridge_role_arn, that role again for delivery, and the publisher itself.

1. The role trusts EventBridge Scheduler to assume it:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "scheduler.amazonaws.com" },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": { "aws:SourceAccount": "123456789012" },
        "ArnLike": {
          "aws:SourceArn": "arn:aws:scheduler:us-east-1:123456789012:schedule/*/*"
        }
      }
    }
  ]
}

2. The role can deliver to your queue(s):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:us-east-1:123456789012:myqueue"
    }
  ]
}

3. The publisher can create schedules and pass the role to EventBridge:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["scheduler:CreateSchedule", "sqs:GetQueueAttributes"],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "arn:aws:iam::123456789012:role/celery-scheduler"
    }
  ]
}

Failure and recovery

A CreateSchedule failure (throttling, a missing role, etc) raises out of Channel._put, which propagates up through producer.publish, so celery's own task_publish_retry covers it with no extra handling needed from this package.

A schedule that fires but cannot deliver, for instance a deleted queue, a revoked role permission, or a KMS error on an encrypted queue, goes to eventbridge_dead_letter_arn if you set one. Setting it is recommended: without it, a schedule that fails to deliver leaves nothing behind, because ActionAfterCompletion='DELETE' removes the schedule once it has run.

Retries

Celery reuses one task id across every retry, but "The rule" above is applied fresh on each publish. A retry with a growing backoff can start under 900s on native DelaySeconds, then move to EventBridge Scheduler once a later retry's countdown passes 900s, with no special handling on your part:

@app.task(bind=True, max_retries=5)
def my_task(self):
    try:
        ...
    except Exception as exc:
        countdown = 300 * (2**self.request.retries)
        raise self.retry(exc=exc, countdown=countdown)

Quotas

Every delay above 900s costs one EventBridge Scheduler CreateSchedule call and one invocation. Check your account's EventBridge Scheduler quotas (schedule count, CreateSchedule rate, invocation rate) against your expected volume before relying on this path at scale.

Development

make install            # uv sync --dev
make test               # unit tests, no broker needed
make up                 # start a local ministack (https://ministack.org)
make test-integration   # integration tests against that ministack
make test-all           # integration + unit tests
make down
make lint

License

MIT, see LICENSE.