Skip to main content

Threading and Multiprocessing for every project.

Project description

https://raw.githubusercontent.com/BrianPugh/lox/main/assets/lox_200w.png https://img.shields.io/pypi/v/lox.svg https://circleci.com/gh/BrianPugh/lox.svg?style=svg Documentation Status

Threading and multiprocessing made easy.

Lox provides decorators and synchronization primitives to quickly add concurrency to your projects.

Installation

pip3 install –user lox

Features

  • Multithreading: Powerful, intuitive multithreading in just 2 additional lines of code.

  • Multiprocessing: Truly parallel function execution with the same interface as multithreading.

  • Synchronization: Advanced thread synchronization, communication, and resource management tools.

Todos

  • All objects except lox.process are for threads. These will eventually be multiprocess friendly.

Usage

Easy Multithreading

>>> import lox
>>>
>>> @lox.thread(4) # Will operate with a maximum of 4 threads
... def foo(x,y):
...     return x*y
>>> foo(3,4) # normal function calls still work
12
>>> for i in range(5):
...     foo.scatter(i, i+1)
-ignore-
>>> # foo is currently being executed in 4 threads
>>> results = foo.gather() # block until results are ready
>>> print(results) # Results are in the same order as scatter() calls
[0, 2, 6, 12, 20]

Or, for example, if you aren’t allowed to directly decorate the function you would like multithreaded/multiprocessed, you can just directly invoke the decorator:

>>> # Lets say we don't have direct access to this function
... def foo(x, y):
...     return x * y
...
>>>
>>> def my_func():
...     foo_threaded = lox.thread(foo)
...     for i in range(5):
...         foo_threaded.scatter(i, i + 1)
...     results = foo_threaded.gather()
...     # foo is currently being executed in default 50 thread executor pool
...     return results
...

This also makes it easier to dynamically control the number of thread/processes in the executor pool. The syntax is a little weird, but this is just explicitly invoking a decorator that has optional arguments:

>>> # Set the number of executer threads to 10
>>> foo_threaded = lox.thread(10)(foo)

Easy Multiprocessing

>>> import lox
>>>
>>> @lox.process(4)  # Will operate with a pool of 4 processes
... def foo(x, y):
...     return x * y
...
>>> foo(3, 4)  # normal function calls still work
12
>>> for i in range(5):
...     foo.scatter(i, i + 1)
...
-ignore-
>>> # foo is currently being executed in 4 processes
>>> results = foo.gather()  # block until results are ready
>>> print(results)  # Results are in the same order as scatter() calls
[0, 2, 6, 12, 20]

Progress Bar Support (tqdm)

>>> import lox
>>> from random import random
>>> from time import sleep
>>>
>>> @lox.thread(2)
... def foo(multiplier):
...     sleep(multiplier * random())
...
>>> for i in range(10):
>>> foo.scatter(i)
>>> results = foo.gather(tqdm=True)
90%|████████████████████████████████▌        | 9/10 [00:03<00:00,  1.32it/s]
100%|███████████████████████████████████████| 10/10 [00:06<00:00,  1.46s/it]

History

0.10.0 (2021-12-18)

  • Remove dependency pinning.

  • Allow @lox.thread(0). This will execute scatter calls in parent thread. Useful for debugging breakpoints in parallelized code.

0.9.0 (2020-11-25)

  • tqdm support on lox.process.gather. See v0.8.0 release notes for usage.

0.8.0 (2020-11-25)

  • tqdm support on lox.thread.gather * Can be a bool:

    >>> my_func.gather(tqdm=True)
    • Can be a tqdm object:

      >>> from tqdm import tqdm
      >>> pbar = tqdm(total=100)
      >>> for _ in range(100):
      >>>     my_func.scatter()
      >>> my_func.gather(tqdm=pbar)

0.7.0 (2020-07-20)

  • Complete rework of workers + Fix memory leaks

  • Drop support for python3.5

  • Drop support for chaining in favor of simpler codebase

0.6.3 (2019-07-30)

  • Alternative fix for 0.6.2.

0.6.2 (2019-07-21)

  • Update dependencies

  • Fix garbage-collecting exclusiviity

0.6.1 (2019-07-21)

  • Fix memory leak in lox.process.

0.6.0 (2019-07-21)

  • lox.Announcement subscribe() calls now return another Announcement object that behaves like a queue instead of an actual queue. Allows for many-queue-to-many-queue communications.

  • New Object: lox.Funnel. allows for waiting on many queues for a complete set of inputs indicated by a job ID.

0.5.0 (2019-07-01)

  • New Object: lox.Announcement. Allows a one-to-many thread queue with backlog support so that late subscribers can still get all (or most recent) announcements before they subscribed.

  • New Feature: lox.thread scatter calls can now be chained together. scatter now returns an int subclass that contains metadata to allow chaining. Each scatter call can have a maximum of 1 previous scatter result.

  • Documentation updates, theming, and logos

0.4.3 (2019-06-24)

  • Garbage collect cached decorated object methods

0.4.2 (2019-06-23)

  • Fixed multiple instances and successive scatter and gather calls to wrapped methods

0.4.1 (2019-06-23)

  • Fixed broken workers and unit tests for workers

0.4.0 (2019-06-22)

  • Semi-breaking change: lox.thread and lox.process now automatically pass the object instance when decorating a method.

0.3.4 (2019-06-20)

  • Print traceback in red when a thread crashes

0.3.3 (2019-06-19)

  • Fix bug where thread in scatter of lox.thread double releases on empty queue

0.3.2 (2019-06-17)

  • Fix manifest for installation from wheel

0.3.1 (2019-06-17)

  • Fix package on pypi

0.3.0 (2019-06-01)

  • Multiprocessing decorator. lox.pool renamed to lox.thread

  • Substantial pytest bug fixes

  • Documentation examples

  • timeout for RWLock

0.2.1 (2019-05-25)

  • Fix IndexSemaphore context manager

0.2.0 (2019-05-24)

  • Added QLock

  • Documentation syntax fixes

0.1.1 (2019-05-24)

  • CICD test

0.1.0 (2019-05-24)

  • First release on PyPI.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lox-0.11.0.tar.gz (37.0 kB view hashes)

Uploaded source

Built Distribution

lox-0.11.0-py2.py3-none-any.whl (24.7 kB view hashes)

Uploaded py2 py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page