jasyncq
Asynchronous task queue using mysql
You should know
- Dispatcher's
fetch_scheduled_tasksandfetch_pending_tasksmethod takes scheduled job and concurrently update their status asWORK IN PROGRESSin same transaction - Most of tasks that queued in jasyncq would run in
exactly oncebyfetch_scheduled_tasksBUT, some cases job disappeared because of worker shutdown while working. It could be restored byfetch_pending_tasks(that can check how long worker tolerateWIP-ed but notCompleted(deleted row))
How to use
1. Create aiomysql connection pool
import asyncio
import logging
import aiomysql
loop = asyncio.get_event_loop()
pool = await aiomysql.create_pool(
host='127.0.0.1',
port=3306,
user='root',
db='test',
loop=loop,
autocommit=False,
)
2. Generate topic (table) with initialize and inject repository to dispatcher
from jasyncq.dispatcher.tasks import TasksDispatcher
from jasyncq.repository.tasks import TaskRepository
repository = TaskRepository(pool=pool, topic_name='test_topic')
await repository.initialize()
dispatcher = TasksDispatcher(repository=repository)
3. Enjoy queue
- Publish tasks
await dispatcher.apply_tasks(
tasks=[...list of jasyncq.dispatcher.model.task.TaskIn...],
)
- Consume tasks
scheduled_tasks = await dispatcher.fetch_scheduled_tasks(queue_name='QUEUE_TEST', limit=10)
pending_tasks = await dispatcher.fetch_pending_tasks(
queue_name='QUEUE_TEST',
limit=10,
check_term_seconds=60,
)
tasks = [*pending_tasks, *scheduled_tasks]
# ...RUN JOBS WITH tasks
4. Complete tasks
task_ids = [str(task.uuid) for task in tasks]
await dispatcher.complete_tasks(task_ids=task_ids)
Other features
Apply tasks with dependency
genesis = TaskIn(task={}, queue_name=queue_name)
dependent = TaskIn(task={}, queue_name=queue_name, depend_on=task.uuid)
# 'dependent' task might fetched after 'genesis' task is completed
await dispatcher.apply_tasks(tasks=[genesis, dependent])
Apply delayed task(scheduled task)
scheduled_at = time.time() + 60
task = TaskIn(task={}, queue_name=queue_name, scheduled_at=scheduled_at)
# 'task' task might fetched after 60 seconds from now
await dispatcher.apply_tasks(tasks=[task])
Apply urgent task (priority)
normal = TaskIn(task={}, queue_name=queue_name)
urgent = TaskIn(task={}, queue_name=queue_name, is_urgent=True)
# 'urgent' task might fetched earlier than 'normal' task if queue was already fulled
await dispatcher.apply_tasks(tasks=[normal, urgent])
Fetching with ignoring dependency
scheduled_tasks = await dispatcher.fetch_scheduled_tasks(
queue_name='QUEUE_TEST',
limit=10,
ignore_dependency=True,
)
pending_tasks = await dispatcher.fetch_pending_tasks(
queue_name='QUEUE_TEST',
limit=10,
check_term_seconds=60,
ignore_dependency=True,
)
tasks = [*pending_tasks, *scheduled_tasks]
# ...RUN JOBS WITH tasks
Example
- Consumer: /example/consumer.py
- Producer: /example/producer.py
Run example scripts
$ docker run --name test_db -p 3306:3306 -e MYSQL_ALLOW_EMPTY_PASSWORD=true -d mysql:8.0.17
$ docker exec -it test_db bash -c 'mysql -u root -e "create database test;"'
$ python3 -m example.producer
$ python3 -m example.consumer
Build
$ python3 setup.py sdist
$ python3 -m pip install ./dist/jasyncq-*
Deploy
$ twine upload ./dist/jasyncq-{version}.tar.gz
Test
$ docker run --name test_db -p 3306:3306 -e MYSQL_ALLOW_EMPTY_PASSWORD=true -d mysql:8.0.17
$ docker exec -it test_db bash -c 'mysql -u root -e "create database test;"'
$ python3 -m pip install pytest==6.2.3
$ pytest
Metadata
Release files for jasyncq 1.1.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| jasyncq-1.1.2.tar.gz | 9.1 kB | Without provenance | Details for jasyncq-1.1.2.tar.gz |
Release files / jasyncq-1.1.2.tar.gz
| Download URL | jasyncq-1.1.2.tar.gz |
|---|---|
| Size | 9.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
5c94687ce8f1b2d202663656b14578c534cd49df44e1665ea00bedd6877ef7d5
|
|
BLAKE2b-256 checksum How to use checksums |
3a2bc876a8d5bc7fe9a84d346f06ecc2c5b24af4010854e444fe880104b159c9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/46.1.1 requests-toolbelt/0.9.1 tqdm/4.41.0 CPython/3.7.5
|
Security for jasyncq 1.1.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.