Hades is an multithreaded asynchronous job processing class. It allows you to register different job types and associate processor functions or classes with those job types.
Job processor function needs to expect job data as input. If a job processor is a class, it needs to have run() method that expects job data as input.
Jobs are discovered by watching a folder on the file system for new files. If a file is correctly formated it is parsed and processed passing the parsed data to the registered processor.
A file needs to be a json document with a ‘type’ key matching one of the registered job types.
When the class is initialized it expects a path to the folder to watch. The folder needs to have three subfolders: ‘in’ for incoming jobs, ‘cur’ for storing successfully processed jobs (this can be disabled by passing save_successful=False to the class), and ‘err’ for storing failed jobs (this can be disabled by passing save_failed=False to the class).
By default a pool of 5 processing threads will be started. This can be tuned by passing threads=N to the class.
If a job processing fails, it will be reprocessed few more times as defined by retries attribute (default is 3). If this is not desired it can be disabled by passing retries=0 to the class.
Usage
The usage is very simple. You need to define processors, initialize Hades with the path to the folder to watch, register processors with Hades and call start() method to start all the threads. By default Hades runs in interactive mode. If you want to run it in a daemon mode, just pass ‘daemon=True’ to the start() method.
import hades
class Download:
def run(self, task):
print("Downloading page: {0}".format(task['url']))
return True
def send_email(task):
print("Email for: {0}".format(task['rcpt']))
return True
if __name__ == '__main__':
hades = hades.Hades('/tmp/jobs')
hades.register({'email': send_email})
hades.register({'download': Download})
hades.start(daemon=True)
To send jobs, just deserialize a json object containing the data for your jobs and save them into the defined folder. Hades will pick it up from there.
import json
email = {'type': 'email',
'from': 'no@mail.com',
'rcpt': 'test@example.com',
'subject': 'Test email',
'body': 'Hi there!'}
download = {'type': 'download',
'url': 'http://www.miljan.org/',
'file': '/tmp/miljan.org.html'}
json.dump(email, open("/tmp/jobs/in/email", "w"))
json.dump(download, open("/tmp/jobs/in/download", "w"))
1 maintainer
Metadata
1 maintainer
Release files for hades 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| hades-0.2.0.tar.gz | 4.5 kB | Without provenance | Details for hades-0.2.0.tar.gz |
Release files / hades-0.2.0.tar.gz
| Download URL | hades-0.2.0.tar.gz |
|---|---|
| Size | 4.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
422dc00a1691fdc263784688b3060a768ef14f4382f1647b88403cdee2946406
|
|
BLAKE2b-256 checksum How to use checksums |
f272dc3dfa52e9c933c4dc95017978d8281d2b132d5a2d4a9f685ce742a70ec9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Security for hades 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
