A reliable distributed scheduler with pluggable storage backends
A reliable distributed scheduler with pluggable storage backends for Async Python.
Free software: MIT license
Minimal installation (just SQLite persistence):
pip install pyncette
Full installation (all the backends and Prometheus metrics exporter):
pip install pyncette[all]
You can also install the in-development version with:
pip install https://github.com/tibordp/pyncette/archive/master.zip
Simple in-memory scheduler (does not persist state)
from pyncette import Pyncette, Context app = Pyncette() @app.task(schedule='* * * * *') async def foo(context: Context): print('This will run every minute') if __name__ == '__main__': app.main()
Persistent distributed cron using Redis (coordinates execution with parallel instances and survives restarts)
from pyncette import Pyncette, Context from pyncette.redis import redis_repository app = Pyncette(repository_factory=redis_repository, redis_url='redis://localhost') @app.task(schedule='* * * * * */10') async def foo(context: Context): print('This will run every 10 seconds') if __name__ == '__main__': app.main()
See the examples directory for more examples of usage.
Pyncette is designed for reliable (at-least-once or at-most-once) execution of recurring tasks (think cronjobs) whose lifecycles are managed dynamically, but can work effectively for non-reccuring tasks too.
Example use cases:
You want to perform a database backup every day at noon
You want a report to be generated daily for your 10M users at the time of their choosing
You want currency conversion rates to be refreshed every 10 seconds
You want to allow your users to schedule non-recurring emails to be sent at an arbitrary time in the future
Pyncette might not be a good fit if:
You want your tasks to be scheduled to run (ideally) once as soon as possible. It is doable, but you will be better served by a general purpose reliable queue like RabbitMQ or Amazon SQS.
You need tasks to execute at sub-second intervals with low jitter. Pyncette coordinates execution on a per task-instance basis and this corrdination can add overhead and jitter.
Pyncette comes with an implementation for the following backends (used for persistence and coordination) out-of-the-box:
Redis (pip install pyncette[redis])
PostgreSQL (pip install pyncette[postgres])
MySQL 8.0+ (pip install pyncette[mysql] - [does not work on Python 3.10](https://github.com/aio-libs/aiomysql/issues/624))
Amazon DynamoDB (pip install pyncette[dynamodb])
Pyncette imposes few requirements on the underlying datastores, so it can be extended to support other databases or custom storage formats / integrations with existing systems. For best results, the backend needs to provide:
Some sort of serialization mechanism, e.g. traditional transactions, atomic stored procedures or compare-and-swap
Efficient range queries over a secondary index, which can be eventually consistent
To run integration tests you will need Redis, PostgreSQL, MySQL and Localstack (for DynamoDB) running locally.
To run the all tests run:
Alternatively, there is a Docker Compose environment that will set up all the backends so that integration tests can run seamlessly:
# If you want the Docker environment to run as current user # to avoid writing files as root. export UID_GID="$(id -u):$(id -g)" docker-compose up -d docker-compose run --rm shell tox
To run just the unit tests (excluding integration tests):
tox -e py38 # or py37, py39
Note, to combine the coverage data from all the tox environments run:
set PYTEST_ADDOPTS=--cov-append tox
Improve performance for calculation of the next execution time
Add ability for repositories to pass a pagination token
Add add_to_context() to inject static data to context
Clean up documentation and add additional examples
Added Amazon DynamoDB backend
Added MySQL backend
Added support for partitioned dynamic tasks
Added support for automatic and cooperative lease heartbeating
PostgreSQL backend can now skip automatic table creation
Improved signal handling
CI: Add Codecov integration
Devenv: Run integration tests in Docker Compose
Optimize the task querying on Postgres backend
Fix: ensure that there are no name colissions between concrete instances of different dynamic tasks
Improve fairness of polling tasks under high contention.
Added PostgreSQL backend
Added Sqlite backend and made it the default (replacing InMemoryRepository)
Refactored test suite to cover all conformance/integration tests on all backends
Refactored Redis backend, simplifying the Lua scripts and improving exceptional case handling (e.g. tasks disappearing between query and poll)
Main loop only sleeps for the rest of remaining poll_interval before next tick instead of the full amount
General bug fixes, documentation changes, clean up
Fixes bug where a locked dynamic task could be executed again on next tick.
poll_task is now reentrant with regards to locking. If the lease passed in matches the lease on the task, it behaves as though it were unlocked.
Middleware support and optional metrics via Prometheus
Improved the graceful shutdown behavior
Task instance and application context are now available in the task context
Breaking change: dynamic task parameters are now accessed via context.args[‘name’] instead of context.name
Improved examples, documentation and packaging
More efficient poling when Redis backend is used
First release that actually works.
First release on PyPI.
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Hashes for pyncette-0.9.0-py2.py3-none-any.whl