Skip to main content

A reliable distributed scheduler with pluggable storage backends

Project description

A reliable distributed scheduler with pluggable storage backends

  • Free software: MIT license

Installation

Minimal installation (just SQLite persistence):

pip install pyncette

Full installation (Redis and PostgreSQL persistence and Prometheus metrics exporter):

pip install pyncette[redis,postgres,prometheus]

You can also install the in-development version with:

pip install https://github.com/tibordp/pyncette/archive/master.zip

Usage example

Simple in-memory scheduler (does not persist state)

from pyncette import Pyncette, Context

app = Pyncette()

@app.task(schedule='* * * * *')
async def foo(context: Context):
    print('This will run every minute')

if __name__ == '__main__':
    app.main()

Persistent distributed cron using Redis (coordinates execution with parallel instances and survives restarts)

from pyncette import Pyncette, Context
from pyncette.redis import redis_repository

app = Pyncette(repository_factory=redis_repository, redis_url='redis://localhost')

@app.task(schedule='* * * * * */10')
async def foo(context: Context):
    print('This will run every 10 seconds')

if __name__ == '__main__':
    app.main()

See the examples directory for more examples of usage.

Development

To run integration tests you will need Redis and PostgreSQL Server running locally.

To run the all tests run:

tox

To run just the unit tests (excluding integration tests):

tox -e py37  # or py38

Note, to combine the coverage data from all the tox environments run:

Windows
set PYTEST_ADDOPTS=--cov-append
tox
Other
PYTEST_ADDOPTS=--cov-append tox

Changelog

0.6.1 (2020-04-02)

  • Optimize the task querying on Postgres backend
  • Fix: ensure that there are no name colissions between concrete instances of different dynamic tasks
  • Improve fairness of polling tasks under high contention.

0.6.0 (2020-03-31)

  • Added PostgreSQL backend
  • Added Sqlite backend and made it the default (replacing InMemoryRepository)
  • Refactored test suite to cover all conformance/integration tests on all backends
  • Refactored Redis backend, simplifying the Lua scripts and improving exceptional case handling (e.g. tasks disappearing between query and poll)
  • Main loop only sleeps for the rest of remaining poll_interval before next tick instead of the full amount
  • General bug fixes, documentation changes, clean up

0.5.0 (2020-03-27)

  • Fixes bug where a locked dynamic task could be executed again on next tick.
  • poll_task is now reentrant with regards to locking. If the lease passed in matches the lease on the task, it behaves as though it were unlocked.

0.4.0 (2020-02-16)

  • Middleware support and optional metrics via Prometheus
  • Improved the graceful shutdown behavior
  • Task instance and application context are now available in the task context
  • Breaking change: dynamic task parameters are now accessed via context.args[‘name’] instead of context.name
  • Improved examples, documentation and packaging

0.2.0 (2020-01-08)

  • Timezone support
  • More efficient poling when Redis backend is used

0.1.1 (2020-01-08)

  • First release that actually works.

0.0.0 (2019-12-31)

  • First release on PyPI.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pyncette, version 0.6.3
Filename, size File type Python version Upload date Hashes
Filename, size pyncette-0.6.3-py2.py3-none-any.whl (25.9 kB) File type Wheel Python version py2.py3 Upload date Hashes View
Filename, size pyncette-0.6.3.tar.gz (43.0 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page