unlockedpd
Unlock pandas performance with zero code changes.
unlockedpd is a drop-in performance booster for pandas that achieves 5-15x speedups on rolling, expanding, EWM, and cumulative operations. Just import unlockedpd after pandas and your existing code runs faster.
import pandas as pd
import unlockedpd # That's it. Your pandas code is now faster.
df = pd.DataFrame(...)
df.rolling(20).mean() # 5x faster!
df.expanding().max() # 15x faster!
df.ewm(span=20).mean() # 4.8x faster!
Why unlockedpd?
| Library | Speedup | pandas Compatible | Setup Required |
|---|---|---|---|
| unlockedpd | 8.7x avg | 100% | pip install |
| Polars | 5-10x | 0% (new API) | Learn new API |
| Modin | ~4x | 95% | Ray/Dask cluster |
Key advantages:
- Zero code changes: Works with your existing pandas code
- No infrastructure: No Ray, no Dask, no distributed setup
- No new API to learn: It's still pandas
- Automatic fallback: Falls back to pandas for unsupported cases
Benchmarks
Tested on a 64-core machine with a 0.8GB DataFrame (10,000 rows x 10,000 columns):
Rolling Operations (8.4x average)
| Operation | pandas | unlockedpd | Speedup |
|---|---|---|---|
rolling(20).mean() |
1.96s | 0.39s | 5.0x |
rolling(20).sum() |
1.78s | 0.18s | 9.7x |
rolling(20).std() |
2.51s | 0.40s | 6.3x |
rolling(20).var() |
2.36s | 0.40s | 5.9x |
rolling(20).min() |
3.30s | 0.28s | 11.6x |
rolling(20).max() |
3.36s | 0.29s | 11.6x |
Expanding Operations (10.7x average)
| Operation | pandas | unlockedpd | Speedup |
|---|---|---|---|
expanding().mean() |
1.55s | 0.20s | 7.9x |
expanding().sum() |
1.46s | 0.18s | 8.3x |
expanding().std() |
1.89s | 0.20s | 9.6x |
expanding().var() |
1.65s | 0.18s | 9.1x |
expanding().min() |
2.61s | 0.18s | 14.3x |
expanding().max() |
2.69s | 0.18s | 15.1x |
EWM Operations (5.3x average)
| Operation | pandas | unlockedpd | Speedup |
|---|---|---|---|
ewm(span=20).mean() |
1.18s | 0.25s | 4.8x |
ewm(span=20).std() |
1.51s | 0.37s | 4.0x |
ewm(span=20).var() |
1.31s | 0.19s | 7.1x |
Cumulative Operations (3.2x average)
| Operation | pandas | unlockedpd | Speedup |
|---|---|---|---|
cumsum() |
0.59s | 0.19s | 3.2x |
cummin() |
0.58s | 0.18s | 3.2x |
cummax() |
0.58s | 0.19s | 3.1x |
Other Operations
| Operation | Speedup |
|---|---|
pct_change() |
11x |
rank(axis=1) |
8-10x |
rank(axis=0) |
1.4-1.5x |
diff() |
1.0-1.7x |
shift() |
1.0-1.5x |
Installation
pip install unlockedpd
Requirements:
- Python 3.9+
- pandas >= 1.5
- numba >= 0.56
- numpy >= 1.21
Usage
Basic Usage
import pandas as pd
import unlockedpd # Import after pandas
# Your existing code works unchanged
df = pd.DataFrame(np.random.randn(10000, 1000))
result = df.rolling(20).mean() # Automatically optimized!
Configuration
import unlockedpd
# Disable optimizations temporarily
unlockedpd.config.enabled = False
# Set thread count (default: min(cpu_count, 32))
unlockedpd.config.num_threads = 16
# Enable warnings when falling back to pandas
unlockedpd.config.warn_on_fallback = True
# Set minimum elements for parallel execution
unlockedpd.config.parallel_threshold = 500_000
Environment Variables
export UNLOCKEDPD_ENABLED=false
export UNLOCKEDPD_NUM_THREADS=16
export UNLOCKEDPD_WARN_ON_FALLBACK=true
export UNLOCKEDPD_PARALLEL_THRESHOLD=500000
Temporarily Disable
from unlockedpd import _PatchRegistry
with _PatchRegistry.temporarily_unpatched():
# Uses original pandas here
result = df.rolling(20).mean()
How It Works
unlockedpd achieves its speedups through:
- Numba JIT compilation: Operations are compiled to optimized machine code
nogil=True: Releases Python's GIL during computation- ThreadPoolExecutor: Achieves true parallelism across CPU cores
- Column-wise chunking: Distributes work efficiently across threads
The key insight: @njit(nogil=True) + ThreadPoolExecutor combines Numba's fast compiled loops with true multi-threaded parallelism.
┌─────────────────────────────────────────────────────────────┐
│ ThreadPoolExecutor │
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
│ │ Thread 1│ │ Thread 2│ │ Thread 3│ ... │Thread 32│ │
│ │ Cols 0-k│ │Cols k-2k│ │Cols 2k..│ │Cols ..N │ │
│ │ (nogil) │ │ (nogil) │ │ (nogil) │ │ (nogil) │ │
│ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │
└─────────────────────────────────────────────────────────────┘
What's Optimized
Fully optimized (5-15x faster):
rolling().mean(),sum(),std(),var(),min(),max(),count(),skew(),kurt(),median(),quantile()expanding().mean(),sum(),std(),var(),min(),max(),count(),skew(),kurt()ewm().mean(),std(),var()cumsum(),cumprod(),cummin(),cummax()rank()(both axis=0 and axis=1)pct_change(),diff(),shift()rolling().corr(),rolling().cov()(pairwise)
Passes through to pandas (unchanged):
rolling().apply()(custom functions)- Series operations (optimizations target DataFrames)
- Non-numeric columns (auto-fallback)
Compatibility
unlockedpd is designed for 100% pandas compatibility:
- Drop-in replacement: No code changes required
- Automatic fallback: If optimization fails, falls back to pandas
- Type preservation: Returns same types as pandas
- Index preservation: Maintains DataFrame/Series indices
- NaN handling: Correctly handles missing values
Comparison with Alternatives
vs Polars
| Aspect | unlockedpd | Polars |
|---|---|---|
| Speedup | 8.7x avg | 5-10x |
| API | pandas (unchanged) | New API to learn |
| Code changes | None | Rewrite required |
| Ecosystem | pandas ecosystem | Polars ecosystem |
vs Modin
| Aspect | unlockedpd | Modin |
|---|---|---|
| Speedup | 8.7x avg | ~4x (general) |
| Rolling ops | 8.4x optimized | Not optimized |
| Infrastructure | None | Ray/Dask cluster |
| Memory | Low overhead | Partitioning overhead |
vs Vanilla Numba
| Aspect | unlockedpd | Manual Numba |
|---|---|---|
| Usage | import unlockedpd |
Write custom kernels |
| GIL handling | Automatic (nogil=True) |
Manual |
| Parallelization | Automatic ThreadPool | Manual implementation |
Running Benchmarks
# Clone the repo
git clone https://github.com/Yeachan-Heo/unlockedpd
cd unlockedpd
# Install with dev dependencies
pip install -e ".[dev]"
# Run benchmarks
pytest benchmarks/ -v
Contributing
Contributions are welcome! Areas of interest:
- Additional operation optimizations
- Performance improvements
- Documentation and examples
- Bug reports and fixes
Changelog
v0.2.2 (2026-01-21)
100% Pandas Compatibility Fixes:
- Fixed
pct_change()division by zero - Now correctly returnsinf/-infwhen dividing by zero (previously returnedNaN) - Fixed rolling
skew()andkurt()- Corrected bias correction formulas to match pandas exactly - Fixed expanding
skew()andkurt()- Corrected bias correction formulas to match pandas exactly - Fixed zero variance handling - Rolling/expanding skew returns
0.0, kurt returns-3.0for constant data (matching pandas) - Fixed EWM
mean()formula - Corrected weight decay calculation to match pandas exactly - Fixed EWM
std()andvar()- Corrected bias correction formula and first value handling (returns NaN for first observation)
All operations now pass strict pandas compatibility tests (rtol=1e-10) for edge cases including:
- All-NaN columns
- Zero variance (constant data)
- Near-zero variance (numerical precision edge cases)
- Division by zero in pct_change
v0.2.1 (2026-01-20)
Critical Bug Fix:
- Fixed
pct_change()NaN handling to match pandas default behavior- Previous versions treated
fill_method=Noneas default, causing 5x more NaN values - Now correctly defaults to
fill_method='pad'(forward fill before computing), matching pandas - This fix resolves "Weights are all zero" errors in downstream applications using unlockedpd
- Previous versions treated
API:
pct_change(fill_method='pad')- Default, matches pandas behavior (forward fills NaN before computing)pct_change(fill_method=None)- No fill, fastest option (4.8x vs pandas), use when data has no NaN
v0.2.0 (2026-01-20)
- Major performance improvements across all operations
- Added EWM, expanding, cumulative, and pairwise operations
- Improved parallel dispatch and memory layout optimization
v0.1.0 (2026-01-19)
- Initial release with rolling, rank, and transform operations
License
MIT License - see LICENSE for details.
Acknowledgments
Built with:
How This Project Was Built
This entire project was built using oh-my-claude-sisyphus, an advanced Claude Code harness that enables autonomous, iterative development with specialized AI agents. The codebase, benchmarks, documentation, and optimizations were all generated through the sisyphus workflow orchestration system.
Key oh-my-claude-sisyphus features used:
- Ralph-Plan: Iterative planning with Prometheus (planner), Oracle (advisor), and Momus (reviewer) agents
- Ultrawork Mode: Parallel agent execution for maximum throughput
- Sisyphus-Junior: Focused task execution for implementation work
unlockedpd - Because your pandas code deserves to be fast.
Metadata
Release files for unlockedpd 0.3.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| unlockedpd-0.3.1.tar.gz | 147.4 kB | Without provenance | Details for unlockedpd-0.3.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| unlockedpd-0.3.1-py3-none-any.whl | Python 3 | none | any | Details for unlockedpd-0.3.1-py3-none-any.whl |
Total release size: 214.3 kB
Release files / unlockedpd-0.3.1.tar.gz
| Download URL | unlockedpd-0.3.1.tar.gz |
|---|---|
| Size | 147.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
cf9cabd8eae7880592b607649830489be5edbe13dc2a1be4e3b99e9de26c9625
|
|
BLAKE2b-256 checksum How to use checksums |
d1d16539b5afbad059cda96379fdfa117fe292cfb660b6faa133cca616573bfe
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.10.12
|
Release files / unlockedpd-0.3.1-py3-none-any.whl
| Download URL | unlockedpd-0.3.1-py3-none-any.whl |
|---|---|
| Size | 67.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e8ca71ad548212ed718d671d72b85009fd2579deb4fb50fc069dafe201998ec3
|
|
BLAKE2b-256 checksum How to use checksums |
d22fe94bea2e4cc234998cd003130307dbb9f9f154c8447dc0b06ef5f034b9cc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.10.12
|
Security for unlockedpd 0.3.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.