mpmsub - Memory-Aware Multiprocessing Subprocess
A Python library for running subprocess commands with intelligent memory-aware scheduling and resource management.
Features
- Memory-aware scheduling with automatic resource management
- Pipeline support for chaining commands
- Multiple job interfaces (dictionary, object-oriented, convenience functions)
- Output redirection and progress tracking
- Job profiling for memory optimization
- Flexible API with multiple parameter names
Installation
pip install mpmsub
Quick Start
import mpmsub
# Create cluster and add jobs
p = mpmsub.cluster(cpu=4, memory="8G")
# Dictionary interface
p.jobs.append({"cmd": ["echo", "hello"], "p": 1, "m": "1G"})
# Object interface
p.jobs.append(mpmsub.Job(["python", "script.py"]).cpu(2).memory("2G"))
# Pipeline interface
p.jobs.append(mpmsub.pipeline([
["cat", "data.txt"],
["grep", "pattern"]
], cpu=1, memory="500M"))
# Run and analyze
results = p.run()
print(f"Completed: {results['jobs']['completed']}/{results['jobs']['total']}")
Performance
Run benchmarks to see performance benefits:
python examples/benchmark_demo.py
python benchmarks/run_all_benchmarks.py
Benefits include 1.2-2x speedup in memory-constrained scenarios and better system stability. See benchmarks/README.md for details.
Documentation
Complete documentation: https://nextgenusfs.github.io/mpmsub/
Includes tutorials, API reference, examples, and performance tips.
Key Features
Multiple Job Interfaces
# Dictionary interface
p.jobs.append({"cmd": ["echo", "hello"], "p": 1, "m": "1G"})
# Object interface with builder pattern
job = mpmsub.Job(["python", "script.py"]) \
.cpu(2).memory("4G") \
.stdout_to("output.txt")
# Pipeline interface
pipeline = mpmsub.pipeline([
["cat", "data.txt"],
["grep", "pattern"],
["sort"]
], cpu=1, memory="500M")
Job Profiling
# Measure actual memory usage
profile_results = p.profile()
# Use recommendations for optimized scheduling
p.jobs.append({"cmd": ["my_command"], "p": 1, "m": "150M"})
Memory Formats
"1G"- Gigabytes"1024M"- Megabytes1024- MB (integer)
Examples
See the examples/ directory for usage demonstrations and the documentation for comprehensive tutorials.
Development
# Clone and install in development mode
git clone https://github.com/nextgenusfs/mpmsub.git
cd mpmsub
pip install -e .[dev]
# Install pre-commit hooks (optional)
pre-commit install
CI/CD
The project uses GitHub Actions for:
- Tests: Run on Python 3.8-3.12, Ubuntu/macOS
- Code Quality: Linting, formatting, type checking
- Publishing: Manual workflow for TestPyPI/PyPI, automatic PyPI on releases
- Documentation: Auto-deploy to GitHub Pages
Requirements
- Python 3.8+
- psutil >= 5.8.0
License
MIT License
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file mpmsub-2025.10.16.tar.gz.
File metadata
- Download URL: mpmsub-2025.10.16.tar.gz
- Upload date:
- Size: 20.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
cb72dc2b96ec16cce1ab38b68db13ed7cd15f68e9ac6ab840c109e4d93129a30
|
|
| MD5 |
390258067ef35e1f64cf71c9e1692244
|
|
| BLAKE2b-256 |
cf0f48ea1d3c220c50543bcfbc63d6d55c1eab8efb410d99702530eebe95930e
|
Provenance
The following attestation bundles were made for mpmsub-2025.10.16.tar.gz:
Publisher:
publish.yml on nextgenusfs/mpmsub
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
mpmsub-2025.10.16.tar.gz -
Subject digest:
cb72dc2b96ec16cce1ab38b68db13ed7cd15f68e9ac6ab840c109e4d93129a30 - Sigstore transparency entry: 615546663
- Sigstore integration time:
-
Permalink:
nextgenusfs/mpmsub@4e9481b9bc1e35d632a6c9cc30f2147885d1daad -
Branch / Tag:
refs/heads/main - Owner: https://github.com/nextgenusfs
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish.yml@4e9481b9bc1e35d632a6c9cc30f2147885d1daad -
Trigger Event:
workflow_dispatch
-
Statement type:
File details
Details for the file mpmsub-2025.10.16-py3-none-any.whl.
File metadata
- Download URL: mpmsub-2025.10.16-py3-none-any.whl
- Upload date:
- Size: 17.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
835cc362b03cfb5798f44e5d13f7b1abd195a1963c46d3509c54ff517c804f37
|
|
| MD5 |
909265de490c7255248d87c41e552371
|
|
| BLAKE2b-256 |
d953cc1d935f6e8ae87488ba78c72d0ee0de237897ba0fdd4700d907eb41e7dc
|
Provenance
The following attestation bundles were made for mpmsub-2025.10.16-py3-none-any.whl:
Publisher:
publish.yml on nextgenusfs/mpmsub
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
mpmsub-2025.10.16-py3-none-any.whl -
Subject digest:
835cc362b03cfb5798f44e5d13f7b1abd195a1963c46d3509c54ff517c804f37 - Sigstore transparency entry: 615546734
- Sigstore integration time:
-
Permalink:
nextgenusfs/mpmsub@4e9481b9bc1e35d632a6c9cc30f2147885d1daad -
Branch / Tag:
refs/heads/main - Owner: https://github.com/nextgenusfs
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish.yml@4e9481b9bc1e35d632a6c9cc30f2147885d1daad -
Trigger Event:
workflow_dispatch
-
Statement type:
