parfun

Citi/Parfun

Lightweight parallelization library for Python.

Parfun is a lightweight library making it easy to write and run Python in parallel and distributed systems.

The main feature of the library is its @parallel decorator that transparently executes standard Python functions in parallel following the map-reduce pattern:

from typing import List

import parfun as pf


@pf.parallel(
    # parallelize by chunking the argument list (map)
    split=pf.per_argument(
        values=pf.py_list.by_chunk
    ),

    # merge the output by concatenating the results (reduce)
    combine_with=pf.py_list.concat,
)
def list_pow(values: List[float], factor: float) -> List[float]:
    """compute powers of a list of numbers"""
    return [v**factor for v in values]


if __name__ == "__main__":
    with pf.set_parallel_backend_context("local_multiprocessing"):  # use a local pool of processes
        print(list_pow([1, 2, 3], 2))  # runs in parallel, prints [1, 4, 9]

Features

Provides significant speedups to existing Python functions.
Only requires basic understanding of parallel and distributed computing systems.
Automatically estimates the optimal sub-task splitting strategy (the partition size).
Transparently handles data transmission, caching, and synchronization.
Supports various distributed computing backends:
- Python's built-in multiprocessing module.
- Scaler.
- Dask.

Quick Start

Install Parfun directly from PyPI:

pip install parfun
pip install "parfun[pandas,scaler,dask]"  # with optional dependencies

The official documentation is available at citi.github.io/parfun/.

Take a look at our documentation's quickstart tutorial to get more examples and a deeper overview of the library.

Alternatively, you can build the documentation from source:

cd docs
pip install -r requirements.txt
make html

The documentation's main page can then be found at docs/build/html/index.html.

Benchmarks

Parfun effectively parallelizes even short-duration functions.

For example, when running a short 0.28-second machine learning function on an AMD Epyc 7313 16-Core Processor, we found that Parfun provided an impressive 7.4x speedup. Source code for this experiment here.

Benchmark Results

Contributing

Your contributions are at the core of making this a true open source project. Any contributions you make are greatly appreciated.

We welcome you to:

Fix typos or touch up documentation
Share your opinions on existing issues
Help expand and improve our library by opening a new issue

Please review our community contribution guidelines and functional contribution guidelines to get started 👍.

Code of Conduct

We are committed to making open source an enjoyable and respectful experience for our community. See CODE_OF_CONDUCT for more information.

License

This project is distributed under the Apache-2.0 License. See LICENSE for more information.

Contact

If you have a query or require support with this project, raise an issue. Otherwise, reach out to opensource@citi.com.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

parfun-7.2.2.tar.gz (676.7 kB view details)

Uploaded Apr 23, 2025 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

parfun-7.2.2-py3-none-any.whl (44.3 kB view details)

Uploaded Apr 23, 2025 Python 3

File details

Details for the file parfun-7.2.2.tar.gz.

File metadata

Download URL: parfun-7.2.2.tar.gz
Upload date: Apr 23, 2025
Size: 676.7 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.2

File hashes

Hashes for parfun-7.2.2.tar.gz
Algorithm	Hash digest
SHA256	`a60034ced7d911aec8a2e5bf0f2112227118458efab84825ff6c1feb4309396e`
MD5	`5756bc90b0e03cfe9ef254ccb1181aae`
BLAKE2b-256	`d282585d6ccd51a9c2fa50191ea80dcc5496dd7b73506da22fb855053a288468`

See more details on using hashes here.

Provenance

The following attestation bundles were made for parfun-7.2.2.tar.gz:

Publisher: pypi.yml on Citi/parfun

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: parfun-7.2.2.tar.gz
- Subject digest: a60034ced7d911aec8a2e5bf0f2112227118458efab84825ff6c1feb4309396e
- Sigstore transparency entry: 201495297
- Sigstore integration time: Apr 23, 2025
Source repository:
- Permalink: Citi/parfun@ab5d1e910c2e89a6b2d45bfd392fef11bef86716
- Branch / Tag: refs/tags/v7.2.2
- Owner: https://github.com/Citi
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: pypi.yml@ab5d1e910c2e89a6b2d45bfd392fef11bef86716
- Trigger Event: release