Skip to main content

Scalable, high-performance AsyncIO task scheduler

Project description

aioscheduler

aioscheduler is a scalable and high-performance task scheduler for asyncio.

It schedules execution of coroutines at a specific time in a single task, making it lightweight and extremely scalable by adding a manager for multiple schedulers.

Tests have shown that aioscheduler can run up to 10 million timed tasks with up to 20 finishing per second when using 20 schedulers. Single tasks can easily schedule up to 10.000 tasks. This is based on tests on a Xeon E5 1650v3.

Installation

pip install aioscheduler

Usage

aioscheduler provides several Scheduler classes that runs a main task to consume coroutines.

There are QueuedScheduler/LifoQueuedScheduler and TimedScheduler, whereas TimedScheduler is the default for Managers.

The TimedScheduler compares datetime objects to UTC by default, to disable it, pass prefer_utc=False to the constructor.

import asyncio
from datetime import datetime, timedelta
from aioscheduler import TimedScheduler

async def work(n: int) -> None:
    print(f"I am doing heavy work: {n}")

async def main() -> None:
    starting_time = datetime.utcnow()
    scheduler = TimedScheduler()
    scheduler.start()

    for i in range(60):
        scheduler.schedule(work(i), starting_time + timedelta(seconds=5 + i))

    await asyncio.sleep(65)

asyncio.run(main())

In this example, 60 tasks are scheduled to run in 5 seconds from now, 1 of them per second over a time of 1 minute.

The QueuedScheduler works identical, but consumes tasks in scheduling order immediately and only takes a single coroutine as argument to schedule().

To scale even further, aioscheduler offers the Manager (example with QueuedScheduler backend):

import asyncio
from datetime import datetime, timedelta
from aioscheduler import Manager, QueuedScheduler

async def work(n: int) -> None:
    print(f"I am doing heavy work: {n}")

async def main() -> None:
    starting_time = datetime.utcnow()
    manager = Manager(5, cls=QueuedScheduler)  # The number of Schedulers to use
                                               # Leaving out cls defaults to TimedScheduler
    manager.start()

    for i in range(30000):
        manager.schedule(work(i))

    await asyncio.sleep(5)

asyncio.run(main())

The manager distributes the tasks across multiple schedulers internally and acts as a load-balancer.

schedule() returns a Task object, you may cancel a task after scheduling by running scheduler.cancel(task) (or manager.cancel(task)). The manager is less efficient for cancelling.

To limit the amount of tasks scheduled, there is a max_tasks argument that takes a positive integer. It is advised to use this in production enviroments of known task queue sizes and available on both Scheduler and Manager.

License

MIT

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

aioscheduler-1.4.2.tar.gz (6.4 kB view details)

Uploaded Source

Built Distribution

aioscheduler-1.4.2-py3-none-any.whl (9.0 kB view details)

Uploaded Python 3

File details

Details for the file aioscheduler-1.4.2.tar.gz.

File metadata

  • Download URL: aioscheduler-1.4.2.tar.gz
  • Upload date:
  • Size: 6.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.8.5

File hashes

Hashes for aioscheduler-1.4.2.tar.gz
Algorithm Hash digest
SHA256 9fb189c77252baf469f1b2f74dc8321f8516bc3a0428b4160314c0df22ebee06
MD5 43ab4ee58b4e9c5fee8c1a5de21bbd81
BLAKE2b-256 6ea1884dcaf4d5370fe5d94f345076474965799ed1b787483cc451ad12ab2ca4

See more details on using hashes here.

File details

Details for the file aioscheduler-1.4.2-py3-none-any.whl.

File metadata

  • Download URL: aioscheduler-1.4.2-py3-none-any.whl
  • Upload date:
  • Size: 9.0 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.1.0 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.8.5

File hashes

Hashes for aioscheduler-1.4.2-py3-none-any.whl
Algorithm Hash digest
SHA256 8aaeb7bc01df82136fca38e77f4fc75a503c1d78d23917e15ee8c540e3e4bc23
MD5 d2a9dc5e2426472a444f1715142b7a37
BLAKE2b-256 f3ccb65daefef43796fec989dc44fe4dddb5fd5bf4e9de4ae3a1a6579457bb68

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page