mapply
mapply provides a sensible multi-core apply function for Pandas.
mapply vs. pandarallel vs. swifter
Where pandarallel relies on in-house multiprocessing and progressbars, and hard-codes max_chunks_per_worker=1 (which will cause idle CPUs when one chunk happens to be more expensive than the others), swifter relies on the heavy dask framework for multiprocessing (converting to Dask DataFrames and back) and also hard-codes chunking behaviour at a 'smart default'. In an attempt to find the golden mean, mapply is highly customizable and remains lightweight, using tqdm for progressbars and leveraging the powerful pathos framework, which shadows Python's built-in multiprocessing module using dill for universal pickling.
Installation
This pure-Python, OS independent package is available on PyPI:
$ pip install mapply
Usage
For documentation, see mapply.readthedocs.io.
import pandas as pd
import mapply
mapply.init(
n_workers=-1,
chunk_size=100,
max_chunks_per_worker=8,
progressbar=False
)
df = pd.DataFrame({"A": list(range(100))})
# avoid unnecessary multiprocessing:
# due to chunk_size=100, this will act as regular apply.
# set chunk_size=1 to skip this check and let max_chunks_per_worker decide.
df["squared"] = df.A.mapply(lambda x: x ** 2)
Development
Run make help for options like installing for development, linting, testing, and building docs.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file mapply-0.1.19.tar.gz.
File metadata
- Download URL: mapply-0.1.19.tar.gz
- Upload date:
- Size: 8.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/4.0.2 CPython/3.11.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
36c6b0526b048073f34cf6f917d2db1abe0ebc5e8e72c3166c19eaa3d79e2c97
|
|
| MD5 |
03bccd45bd4618941047aa9d29e1fb53
|
|
| BLAKE2b-256 |
244c464bb8b72fbd0278ec097a2de6457a1f0250625a8ecd340d396cc5b51b6d
|
File details
Details for the file mapply-0.1.19-py3-none-any.whl.
File metadata
- Download URL: mapply-0.1.19-py3-none-any.whl
- Upload date:
- Size: 8.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/4.0.2 CPython/3.11.2
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
babfff4d598bb2a75a40872162f4037a2bf2a42af28b3eb5221d0e9adfa5a413
|
|
| MD5 |
40b1ed5f79afa4e5623550623655393c
|
|
| BLAKE2b-256 |
f5a9586df28b4c260ccbf5e1bea11979346da3e2a7c9bc93b1ce80c10bf26c23
|
