Skip to main content

Package to monitor and throttle multiple processes or threads.

Project description

mp_throttle

A Python package to monitor and throttle multiprocessing processes.

Use case:

This package can be used to control and monitor Pythons multiprocessing processes or threading threads. The central Throttle class can be used to monitor the current and the mean frequency of processes/threads and/or to throttle them to execute a limited times per second. This can be used to limit the server requests of a multi-process web crawler.

Example:

    import multiprocessing
    import mp_throttle
    import time

    def work(tank):
        while not tank.kill_flag.is_set():
            # Block until 'fuel' is available.
            tank.await_fuel()
            # do something
        return

    # Limit the processes to 4 per 1 second, by providing 4 'fuel' per second.
    throttle = mp_throttle.Throttle(4,1)
    workerpool = multiprocessing.Pool(processes=4, initializer=work, initargs=(throttle,))
    throttle.start()
    time.sleep(5)
    # Stops the processes and return stats:
    runtime, total_processes, mean_frequency, mean_processes_per_second = throttle.stop()
    print("Runtime: {}".format(runtime))
    print("Total: {}".format(total_processes))
    print("Mean frequency: {}".format(mean_frequency))
    print("Processes per second: {}".format(mean_processes_per_second))

Output:

>>> Runtime: 5.027516841888428
>>> Total: 20
>>> Mean frequency: 0.25046865940093993
>>> Processes per second: 3.9925154803469485

Installation:

To install mp_throttle, simply use pip:

pip install mp_throttle

Documentation:

For the full documentation see docs.elpunkt.eu

How to Contribute:

  1. Test mp_throttle and open an issue to report a bug or discuss a feature idea.
  2. Give general feedback on the code and the package structure. Since this is my first python package, I'm sure there is a lot to feedback on:)
  3. Fork the repository and make your changes.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mp_throttle-0.0.5.tar.gz (5.0 kB view details)

Uploaded Source

Built Distribution

mp_throttle-0.0.5-py3-none-any.whl (5.2 kB view details)

Uploaded Python 3

File details

Details for the file mp_throttle-0.0.5.tar.gz.

File metadata

  • Download URL: mp_throttle-0.0.5.tar.gz
  • Upload date:
  • Size: 5.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.11.1 setuptools/40.0.0 requests-toolbelt/0.8.0 tqdm/4.24.0 CPython/3.5.2

File hashes

Hashes for mp_throttle-0.0.5.tar.gz
Algorithm Hash digest
SHA256 9a2ee2de19476d85c52bcfdcb2f8b3e546b69bdfcb7d9262421326b7244c897b
MD5 da13670d68b06caf1aafce8b27e36add
BLAKE2b-256 24b1fe3c125aca3b2bcc9958bbfac7708dcead03de41a936b98f12f44a051d49

See more details on using hashes here.

Provenance

File details

Details for the file mp_throttle-0.0.5-py3-none-any.whl.

File metadata

  • Download URL: mp_throttle-0.0.5-py3-none-any.whl
  • Upload date:
  • Size: 5.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.11.1 setuptools/40.0.0 requests-toolbelt/0.8.0 tqdm/4.24.0 CPython/3.5.2

File hashes

Hashes for mp_throttle-0.0.5-py3-none-any.whl
Algorithm Hash digest
SHA256 9cc8946dd73759b318cd39b44ed68b290379ae45a023fe5dfbce774fe16e0e87
MD5 de4f98208b03eb93d85a4877fb021274
BLAKE2b-256 79559e98d9e1c6122bbf9126d01747f20ffa53d378d17b1ba2e36c97469ee9f3

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page