Stupid simple background worker based on python.
Project description
kerground
Stupid simple background worker based on python.
Quickstart
Install
pip install kerground
Mark your file workers by naming them with _worker.py
prefix
my_worker.py
Kerground will look in *_worker.py
and will consider each function an event.
Functions from *_worker.py
files must be unique.
Import Kerground
, instantiate it and start sending events:
#my_api.py
from kerground import Kerground
ker = Kerground()
@app.route('/some-task')
def long_wait():
id = ker.send('long_task')
return {'id': id}
Event long_task
is a function name from *_worker.py files
#my_worker.py
import time
def long_task():
# heavy workoad, more than a few seconds job
time.sleep(2)
Your api's and workers must be in the same package/directory
root
├── api
│ ├── __init__.py
│ └── my_api.py
└── worker
├── __init__.py
└── my_worker.py
You are free to use any folder structure.
Open 2 cmd/terminal windows in the example directory:
- in one start your api
python3 api/my_api.py
- in the other one type
kerground
API
ker.send('func_name', *func_args, timeout=None, purge=True)
Send event to kerground worker. send
function will return the id of the task sent to the worker.
You have hot reload on your workers by default! (as long you don't change function names)
timeout
: will show in kerground logs a warning if function takes longer than expected;purge
: ifTrue
when function is executed event will be deleted, ifFalse
event will be deleted after aker.get_response(id)
call.
ker.status(id)
Check status of a task with status
. Kerground has the folowing statuses:
- pending - event is added to kerground queue
- running - event is running
- finished - event was executed succesfully
- failed - event failed to be executed
Also you can check at any time the statuses of your tasks without specifing the id's:
ker.pending()
ker.running()
ker.finished()
ker.failed()
Or check all statuses with:
ker.stats()
ker.get_response(id)
Get the response from event (will be None
if event didn't ran yet).
You can see functions collected from *_worker.py
files with:
ker.events
Why
Under the hood kerground uses pickle for serialization of input/output data, a combination of inspect
methods and built-in getattr
function for dynamically calling the "events"
(functions) from *_worker.py
files.
It's resource frendly (it doesn't use RAM to hold queue), easy to use (import kerground, mark your worker files with _worker.py
prefix and you are set), has hot reload for workers (no need to restart workers each time you make a change) works on multiple cores (uses multiprocessing).
Submit any questions/issues you have! Fell free to fork it and improve it!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for kerground-0.0.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8412b28440ec601cd3757eef066062d55b20f87a90aa46807e23fcb1b1522261 |
|
MD5 | d08b9433b7c2ac8c839091bb97150915 |
|
BLAKE2b-256 | e14fa481a7140860ec70e0b91e0c82333ea70dfde4b082f15f81c5df693f01ba |