Multiprocessing application to download and analyze a content of an html pages.
Project description
This is the documentation for the Shaman. Multiprocessing application to combine different singular handlers against one message.
The initial purpose was to create a tool, that:
- would make possible to download and analyze a content of an html pages.
- simple enough to add a new functionality in it.
- hast to be scalable (multiprocessing).
Actual usage can be different from it. There are some spontaneous ideas:
- scanning a mongo collection and parsing documents in parallel
- parsing a lot of lines from multiple huge files, saving the results to any database (depending on the results)
There are three parts in the shaman library:
* stages (actual processors, which do represent some functionality)
* consumer (worker, that run them all in a particular order)
* daemon (run as many as needed workers. Also used as a CLI unstrument.)
All stages are run in a particular order and use the same message object (inside one worker).
The initial purpose was to create a tool, that:
- would make possible to download and analyze a content of an html pages.
- simple enough to add a new functionality in it.
- hast to be scalable (multiprocessing).
Actual usage can be different from it. There are some spontaneous ideas:
- scanning a mongo collection and parsing documents in parallel
- parsing a lot of lines from multiple huge files, saving the results to any database (depending on the results)
There are three parts in the shaman library:
* stages (actual processors, which do represent some functionality)
* consumer (worker, that run them all in a particular order)
* daemon (run as many as needed workers. Also used as a CLI unstrument.)
All stages are run in a particular order and use the same message object (inside one worker).
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
shaman-0.0.4.dev1.zip
(131.5 kB
view hashes)
Built Distribution
Close
Hashes for shaman-0.0.4.dev1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0d82ac1eaec31fa8e00acf8a0a06c87826981b620cba106c585b1a3a28d94426 |
|
MD5 | 55567652aa1c28aac0c26d5e9dc67c92 |
|
BLAKE2b-256 | 43bb550d0ac0863e8838c3d156965e7b94140d47384ec5d232002b4b498fbf11 |