Skip to main content

A reliable distributed scheduler with pluggable storage backends

Project description

A reliable distributed scheduler with pluggable storage backends for Async Python.

  • Free software: MIT license

Installation

Minimal installation (just SQLite persistence):

pip install pyncette

Full installation (all the backends and Prometheus metrics exporter):

pip install pyncette[all]

You can also install the in-development version with:

pip install https://github.com/tibordp/pyncette/archive/master.zip

Documentation

https://pyncette.readthedocs.io

Usage example

Simple in-memory scheduler (does not persist state)

from pyncette import Pyncette, Context

app = Pyncette()

@app.task(schedule='* * * * *')
async def foo(context: Context):
    print('This will run every minute')

if __name__ == '__main__':
    app.main()

Persistent distributed cron using Redis (coordinates execution with parallel instances and survives restarts)

from pyncette import Pyncette, Context
from pyncette.redis import redis_repository

app = Pyncette(repository_factory=redis_repository, redis_url='redis://localhost')

@app.task(schedule='* * * * * */10')
async def foo(context: Context):
    print('This will run every 10 seconds')

if __name__ == '__main__':
    app.main()

See the examples directory for more examples of usage.

Use cases

Pyncette is designed for reliable (at-least-once or at-most-once) execution of recurring tasks (think cronjobs) whose lifecycles are managed dynamically, but can work effectively for non-reccuring tasks too.

Example use cases:

  • You want to perform a database backup every day at noon

  • You want a report to be generated daily for your 10M users at the time of their choosing

  • You want currency conversion rates to be refreshed every 10 seconds

  • You want to allow your users to schedule non-recurring emails to be sent at an arbitrary time in the future

Pyncette might not be a good fit if:

  • You want your tasks to be scheduled to run (ideally) once as soon as possible. It is doable, but you will be better served by a general purpose reliable queue like RabbitMQ or Amazon SQS.

  • You need tasks to execute at sub-second intervals with low jitter. Pyncette coordinates execution on a per task-instance basis and this corrdination can add overhead and jitter.

Supported backends

Pyncette comes with an implementation for the following backends (used for persistence and coordination) out-of-the-box:

  • SQLite (included)

  • Redis (pip install pyncette[redis])

  • PostgreSQL (pip install pyncette[postgres])

  • MySQL 8.0+ (pip install pyncette[mysql] - [does not work on Python 3.10](https://github.com/aio-libs/aiomysql/issues/624))

  • Amazon DynamoDB (pip install pyncette[dynamodb])

Pyncette imposes few requirements on the underlying datastores, so it can be extended to support other databases or custom storage formats / integrations with existing systems. For best results, the backend needs to provide:

  • Some sort of serialization mechanism, e.g. traditional transactions, atomic stored procedures or compare-and-swap

  • Efficient range queries over a secondary index, which can be eventually consistent

Development

To run integration tests you will need Redis, PostgreSQL, MySQL and Localstack (for DynamoDB) running locally.

To run the all tests run:

tox

Alternatively, there is a Docker Compose environment that will set up all the backends so that integration tests can run seamlessly:

# If you want the Docker environment to run as current user
# to avoid writing files as root.
export UID_GID="$(id -u):$(id -g)"
docker-compose up -d
docker-compose run --rm shell
tox

To run just the unit tests (excluding integration tests):

tox -e py38  # or py37, py39

Note, to combine the coverage data from all the tox environments run:

Windows

set PYTEST_ADDOPTS=--cov-append
tox

Other

PYTEST_ADDOPTS=--cov-append tox

Changelog

0.8.1 (2021-04-08)

  • Improve performance for calculation of the next execution time

  • Add ability for repositories to pass a pagination token

  • Add add_to_context() to inject static data to context

  • Clean up documentation and add additional examples

0.8.0 (2021-04-05)

  • Added Amazon DynamoDB backend

  • Added MySQL backend

  • Added support for partitioned dynamic tasks

0.7.0 (2021-03-31)

  • Added support for automatic and cooperative lease heartbeating

  • PostgreSQL backend can now skip automatic table creation

  • Improved signal handling

  • CI: Add Codecov integration

  • Devenv: Run integration tests in Docker Compose

0.6.1 (2020-04-02)

  • Optimize the task querying on Postgres backend

  • Fix: ensure that there are no name colissions between concrete instances of different dynamic tasks

  • Improve fairness of polling tasks under high contention.

0.6.0 (2020-03-31)

  • Added PostgreSQL backend

  • Added Sqlite backend and made it the default (replacing InMemoryRepository)

  • Refactored test suite to cover all conformance/integration tests on all backends

  • Refactored Redis backend, simplifying the Lua scripts and improving exceptional case handling (e.g. tasks disappearing between query and poll)

  • Main loop only sleeps for the rest of remaining poll_interval before next tick instead of the full amount

  • General bug fixes, documentation changes, clean up

0.5.0 (2020-03-27)

  • Fixes bug where a locked dynamic task could be executed again on next tick.

  • poll_task is now reentrant with regards to locking. If the lease passed in matches the lease on the task, it behaves as though it were unlocked.

0.4.0 (2020-02-16)

  • Middleware support and optional metrics via Prometheus

  • Improved the graceful shutdown behavior

  • Task instance and application context are now available in the task context

  • Breaking change: dynamic task parameters are now accessed via context.args[‘name’] instead of context.name

  • Improved examples, documentation and packaging

0.2.0 (2020-01-08)

  • Timezone support

  • More efficient poling when Redis backend is used

0.1.1 (2020-01-08)

  • First release that actually works.

0.0.0 (2019-12-31)

  • First release on PyPI.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyncette-0.9.0.tar.gz (64.0 kB view hashes)

Uploaded source

Built Distribution

pyncette-0.9.0-py2.py3-none-any.whl (37.6 kB view hashes)

Uploaded py2 py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page