Skip to main content

Experimental task class that buffers messages and processes them as a list.

Project description

Celery Batches

https://img.shields.io/pypi/v/celery-batches.svg https://github.com/clokep/celery-batches/actions/workflows/main.yml/badge.svg Documentation Status

Celery Batches provides a Task class that allows processing of multiple Celery task calls together as a list. The buffer of tasks calls is flushed on a timer and based on the number of queued tasks.

Some potential use-cases for batching of task calls include:

  • De-duplicating tasks.

  • Accumulating / only handling the latest task with similar arguments.

  • Bulk inserting / updating of data.

  • Tasks with expensive setup that can run across a range of arguments.

What do I need?

celery-batches version runs on,

  • Python (3.9, 3.10, 3.11, 3.12, 3.13)

  • PyPy (3.10, 3.11)

And is tested with Celery ~= 5.0.

If you’re running an older version of Python, you need to be running an older version of celery-batches, the last version supporting each Python version is listed below:

  • Python 2.7: celery-batches 0.3.

  • Python 3.4: celery-batches 0.2.

  • Python 3.5: celery-batches 0.3.

  • Python 3.6: celery-batches 0.5.

  • Python 3.7: celery-batches 0.7.

  • Python 3.8: celery-batches 0.9.

If you’re running an older version of Celery, you need to be running an older version of celery-batches:

  • Celery < 4.0: Use celery.contrib.batches instead.

  • Celery 4.0 - 4.3: celery-batches 0.3.

  • Celery 4.4: celery-batches 0.7.

  • Celery 5.0 - 5.1: celery-batches 0.9.

History

Celery Batches was distributed as part of Celery (as celery.contrib.batches) until Celery 4.0. This project updates the Batches code to maintain compatiblity with newer versions of Celery and other fixes. See the Changelog for details.

Additionally, this repository includes the full history of the code from celery.contrib.batches, but rewritten to the celery_batches/__init__.py file.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

celery_batches-0.10.tar.gz (8.5 kB view details)

Uploaded Source

Built Distribution

celery_batches-0.10-py3-none-any.whl (8.7 kB view details)

Uploaded Python 3

File details

Details for the file celery_batches-0.10.tar.gz.

File metadata

  • Download URL: celery_batches-0.10.tar.gz
  • Upload date:
  • Size: 8.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.13.2

File hashes

Hashes for celery_batches-0.10.tar.gz
Algorithm Hash digest
SHA256 2e898dab9bf0fed862107069066179278027fe1b87d4c4ee31a3b81713ed7dd3
MD5 56bd50a885ed3aa61a6841f52f3007c2
BLAKE2b-256 511230140c5961bd7b9a71d6e20f4ff0c93b6b96fee738c244b8118ca38276ee

See more details on using hashes here.

File details

Details for the file celery_batches-0.10-py3-none-any.whl.

File metadata

  • Download URL: celery_batches-0.10-py3-none-any.whl
  • Upload date:
  • Size: 8.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.13.2

File hashes

Hashes for celery_batches-0.10-py3-none-any.whl
Algorithm Hash digest
SHA256 108eb8368797bdbc1792335a090600664951719a9a77b72fa3a90eb91e8d2b86
MD5 a07c065a81b88b9881fe6650a89f616b
BLAKE2b-256 15c7609a9877698e9d057076f5d69664e22c72ad15550d6022bd0469d839ae57

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page