Skip to main content

Parallel processing with progress bars

Project description

p_tqdm

PyPI - Python Version PyPI version Downloads Build Status codecov license

p_tqdm makes parallel processing with progress bars easy.

p_tqdm is a wrapper around pathos.multiprocessing and tqdm. Unlike Python's default multiprocessing library, pathos provides a more flexible parallel map which can apply almost any type of function, including lambda functions, nested functions, and class methods, and can easily handle functions with multiple arguments. tqdm is applied on top of pathos's parallel map and displays a progress bar including an estimated time to completion.

Installation

pip install p_tqdm

Example

Let's say you want to add two lists element by element. Without any parallelism, this can be done easily with a Python map.

l1 = ['1', '2', '3']
l2 = ['a', 'b', 'c']

def add(a, b):
    return a + b
    
added = map(add, l1, l2)
# added == ['1a', '2b', '3c']

But if the lists are much larger or the computation is more intense, parallelism becomes a necessity. However, the syntax is often cumbersome. p_tqdm makes it easy and adds a progress bar too.

from p_tqdm import p_map

added = p_map(add, l1, l2)
# added == ['1a', '2b', '3c']
  0%|                                    | 0/3 [00:00<?, ?it/s]
 33%|████████████                        | 1/3 [00:01<00:02, 1.00s/it]
 66%|████████████████████████            | 2/3 [00:02<00:01, 1.00s/it]
100%|████████████████████████████████████| 3/3 [00:03<00:00, 1.00s/it]

p_tqdm functions

Parallel maps

  • p_map - parallel ordered map
  • p_imap - iterator for parallel ordered map
  • p_umap - parallel unordered map
  • p_uimap - iterator for parallel unordered map

Sequential maps

  • t_map - sequential ordered map
  • t_imap - iterator for sequential ordered map

p_map

Performs an ordered map in parallel.

from p_tqdm import p_map

def add(a, b):
    return a + b

added = p_map(add, ['1', '2', '3'], ['a', 'b', 'c'])
# added = ['1a', '2b', '3c']

p_imap

Returns an iterator for an ordered map in parallel.

from p_tqdm import p_imap

def add(a, b):
    return a + b

iterator = p_imap(add, ['1', '2', '3'], ['a', 'b', 'c'])

for result in iterator:
    print(result) # prints '1a', '2b', '3c'

p_umap

Performs an unordered map in parallel.

from p_tqdm import p_umap

def add(a, b):
    return a + b

added = p_umap(add, ['1', '2', '3'], ['a', 'b', 'c'])
# added is an array with '1a', '2b', '3c' in any order

p_uimap

Returns an iterator for an unordered map in parallel.

from p_tqdm import p_uimap

def add(a, b):
    return a + b

iterator = p_uimap(add, ['1', '2', '3'], ['a', 'b', 'c'])

for result in iterator:
    print(result) # prints '1a', '2b', '3c' in any order

t_map

Performs an ordered map sequentially.

from p_tqdm import t_map

def add(a, b):
    return a + b

added = t_map(add, ['1', '2', '3'], ['a', 'b', 'c'])
# added == ['1a', '2b', '3c']

t_imap

Returns an iterator for an ordered map to be performed sequentially.

from p_tqdm import p_imap

def add(a, b):
    return a + b

iterator = t_imap(add, ['1', '2', '3'], ['a', 'b', 'c'])

for result in iterator:
    print(result) # prints '1a', '2b', '3c'

Shared properties

Arguments

All p_tqdm functions accept any number of iterables as input, as long as the number of iterables matches the number of arguments of the function.

To repeat a non-iterable argument along with the iterables, use Python's partial from the functools library. See the example below.

from functools import partial

l1 = ['1', '2', '3']
l2 = ['a', 'b', 'c']

def add(a, b, c=''):
    return a + b + c

added = p_map(partial(add, c='!'), l1, l2)
# added == ['1a!', '2b!', '3c!']

CPUs

All the parallel p_tqdm functions can be passed the keyword num_cpus to indicate how many CPUs to use. The default is all CPUs. num_cpus can either be an integer to indicate the exact number of CPUs to use or a float to indicate the proportion of CPUs to use.

Note that the parallel Pool objects used by p_tqdm are automatically closed when the map finishes processing.

tqdm instance

All the parallel p_tqdm functions can be passed the keyword tqdm to choose a specific flavor of tqdm. By default, this value is taken from tqdm.auto. The tqdm parameter can be used pass p_tqdm output to tqdm.gui, tqdm.tk or any customized subclass of tqdm.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

p_tqdm-1.4.2.tar.gz (6.0 kB view details)

Uploaded Source

File details

Details for the file p_tqdm-1.4.2.tar.gz.

File metadata

  • Download URL: p_tqdm-1.4.2.tar.gz
  • Upload date:
  • Size: 6.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.12.4

File hashes

Hashes for p_tqdm-1.4.2.tar.gz
Algorithm Hash digest
SHA256 0f860c5facd0b0059da39998e55cfc035563f92d85d2f4895ba88a675c3c7529
MD5 9f70db8798a0ecdf0ff79a1cb8533f31
BLAKE2b-256 ea70d4539ed4e2b5e17f170c1c971a98459cbe95eb9ed7e475298ededb56af22

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page