pyncette 0.8.1

A reliable distributed scheduler with pluggable storage backends

A reliable distributed scheduler with pluggable storage backends for Async Python.

• Free software: MIT license

Installation

Minimal installation (just SQLite persistence):

pip install pyncette

Full installation (all the backends and Prometheus metrics exporter):

pip install pyncette[all]

You can also install the in-development version with:

pip install https://github.com/tibordp/pyncette/archive/master.zip

Documentation

https://pyncette.readthedocs.io

Usage example

Simple in-memory scheduler (does not persist state)

from pyncette import Pyncette, Context

app = Pyncette()

@app.task(schedule='* * * * *')
async def foo(context: Context):
    print('This will run every minute')

if __name__ == '__main__':
    app.main()

Persistent distributed cron using Redis (coordinates execution with parallel instances and survives restarts)

from pyncette import Pyncette, Context
from pyncette.redis import redis_repository

app = Pyncette(repository_factory=redis_repository, redis_url='redis://localhost')

@app.task(schedule='* * * * * */10')
async def foo(context: Context):
    print('This will run every 10 seconds')

if __name__ == '__main__':
    app.main()

See the examples directory for more examples of usage.

Use cases

Pyncette is designed for reliable (at-least-once or at-most-once) execution of recurring tasks (think cronjobs) whose lifecycles are managed dynamically, but can work effectively for non-reccuring tasks too.

Example use cases:

• You want to perform a database backup every day at noon

• You want a report to be generated daily for your 10M users at the time of their choosing

• You want currency conversion rates to be refreshed every 10 seconds

• You want to allow your users to schedule non-recurring emails to be sent at an arbitrary time in the future

Pyncette might not be a good fit if:

• You want your tasks to be scheduled to run (ideally) once as soon as possible. It is doable, but you will be better served by a general purpose reliable queue like RabbitMQ or Amazon SQS.

• You need tasks to execute at sub-second intervals with low jitter. Pyncette coordinates execution on a per task-instance basis and this corrdination can add overhead and jitter.

Supported backends

Pyncette comes with an implementation for the following backends (used for persistence and coordination) out-of-the-box:

• SQLite (included)

• Redis (pip install pyncette[redis])

• PostgreSQL (pip install pyncette[postgres])

• MySQL 8.0+ (pip install pyncette[mysql])

• Amazon DynamoDB (pip install pyncette[dynamodb])

Pyncette imposes few requirements on the underlying datastores, so it can be extended to support other databases or custom storage formats / integrations with existing systems. For best results, the backend needs to provide:

• Some sort of serialization mechanism, e.g. traditional transactions, atomic stored procedures or compare-and-swap

• Efficient range queries over a secondary index, which can be eventually consistent

Development

To run integration tests you will need Redis, PostgreSQL, MySQL and Localstack (for DynamoDB) running locally.

To run the all tests run:

tox

Alternatively, there is a Docker Compose environment that will set up all the backends so that integration tests can run seamlessly:

# If you want the Docker environment to run as current user
# to avoid writing files as root.
export UID_GID="$(id -u):$(id -g)"
docker-compose up -d
docker-compose run --rm shell
tox

To run just the unit tests (excluding integration tests):

tox -e py38  # or py37, py39

Note, to combine the coverage data from all the tox environments run:

Windows	set PYTEST_ADDOPTS=--cov-append tox
Other	PYTEST_ADDOPTS=--cov-append tox

Changelog

0.8.1 (2021-04-08)

• Improve performance for calculation of the next execution time

• Add ability for repositories to pass a pagination token

• Add add_to_context() to inject static data to context

• Clean up documentation and add additional examples

0.8.0 (2021-04-05)

• Added Amazon DynamoDB backend

• Added MySQL backend

• Added support for partitioned dynamic tasks

0.7.0 (2021-03-31)

• Added support for automatic and cooperative lease heartbeating

• PostgreSQL backend can now skip automatic table creation

• Improved signal handling

• CI: Add Codecov integration

• Devenv: Run integration tests in Docker Compose

0.6.1 (2020-04-02)

• Optimize the task querying on Postgres backend

• Fix: ensure that there are no name colissions between concrete instances of different dynamic tasks

• Improve fairness of polling tasks under high contention.

0.6.0 (2020-03-31)

• Added PostgreSQL backend

• Added Sqlite backend and made it the default (replacing InMemoryRepository)

• Refactored test suite to cover all conformance/integration tests on all backends

• Refactored Redis backend, simplifying the Lua scripts and improving exceptional case handling (e.g. tasks disappearing between query and poll)

• Main loop only sleeps for the rest of remaining poll_interval before next tick instead of the full amount

• General bug fixes, documentation changes, clean up

0.5.0 (2020-03-27)

• Fixes bug where a locked dynamic task could be executed again on next tick.

• poll_task is now reentrant with regards to locking. If the lease passed in matches the lease on the task, it behaves as though it were unlocked.

0.4.0 (2020-02-16)

• Middleware support and optional metrics via Prometheus

• Improved the graceful shutdown behavior

• Task instance and application context are now available in the task context

• Breaking change: dynamic task parameters are now accessed via context.args[‘name’] instead of context.name

• Improved examples, documentation and packaging

0.2.0 (2020-01-08)

• Timezone support

• More efficient poling when Redis backend is used

0.1.1 (2020-01-08)

• First release that actually works.

0.0.0 (2019-12-31)

• First release on PyPI.