ZooPipe is a lean, ultra-high-performance data processing engine for Python. It leverages a 100% Rust core to handle I/O and orchestration, while keeping the flexibility of Python for schema validation (via Pydantic) and custom data enrichment (via Hooks).
Read the docs for more information.
✨ Key Features
- 🚀 100% Native Rust Engine: The core execution loop, including CSV and JSON parsing/writing, is implemented in Rust for maximum throughput.
- 🔍 Declarative Validation: Use Pydantic models to define and validate your data structures naturally.
- 🪝 Python Hooks: Transform and enrich data at any stage using standard Python functions or classes.
- 🚨 Automated Error Routing: Native support for routing failed records to a dedicated error output.
- 📊 Multiple Format Support: Optimized readers/writers for CSV, JSONL, and SQL databases.
- 🔧 Two-Tier Parallelism: Orchestrate across processes or clusters with Engines (Local, Ray), and scale throughput at the node level with Rust Executors.
- ☁️ Cloud Native: Native S3 support and zero-config distributed execution on Ray clusters.
⚡ Performance & Benchmarks
Why ZooPipe? Because vectorization isn't always the answer.
Tools like Pandas and Polars are incredible for analytical workloads (groupby, sum, joins) where operations can be vectorized in C/Rust. However, real-world Data Engineering often involves "chaotic ETL": messy custom rules, API calls per row, hashing, conditional cleanup, and complex normalization that forcedly drop down to Python loops.
In these "Heavy ETL" scenarios, ZooPipe outperforms Vectorized DataFrames by 3x-8x.
Key Takeaway: ZooPipe's "Python-First Architecture" with parallel streaming (
PipeManager) avoids the serialization overhead that cripples Polars/Pandas when using Python UDFs (map_elements/apply), and uses 97% less RAM.
⚖️ Is this unfair to Pandas/Polars?
Yes and No.
- Unfair: If your workload is purely analytical (e.g.,
GROUP BY,SUM,JOIN), Polars and Pandas will likely destroy ZooPipe because they can use vectorized C/Rust operations on whole columns at once. - Fair: In real-world Data Engineering, many pipelines are "chaotic". They require custom hashing, API calls per row, conditional normalization, or complex Pydantic validation. In these "Python-UDF heavy" scenarios, vectorization breaks down, and ZooPipe shines by orchestrating parallel Python execution efficiently without the DataFrame overhead.
❓ When to use what?
| Use ZooPipe When... | Use Pandas / Polars When... |
|---|---|
| 🏗️ You have complex, custom Python logic per row (hash, clean, validate). | 🧮 You are doing aggregations (SUM, AVG) or Relational Algebra (JOIN, GROUP BY). |
| 🔄 You are processing streaming data or files larger than RAM. | 💾 Your dataset fits comfortably in RAM (or use LazyFrames). |
| 🛡️ You need strict schema validation (Pydantic) and error handling. | 🔬 You are doing data exploration or statistical analysis. |
| 🚀 You want to mix Rust I/O performance with Python flexibility. | ⚡ Your entire pipeline can be expressed in vectorized expressions. |
🚀 Quick Start
Installation
pip install zoopipe
Or using uv:
uv add zoopipe
Or from source (uv recommended):
uv build
uv run maturin develop --release
Simple Example
from pydantic import BaseModel, ConfigDict
from zoopipe import CSVInputAdapter, CSVOutputAdapter, Pipe
class UserSchema(BaseModel):
model_config = ConfigDict(extra="ignore")
user_id: str
username: str
email: str
pipe = Pipe(
input_adapter=CSVInputAdapter("users.csv"),
output_adapter=CSVOutputAdapter("processed_users.csv"),
error_output_adapter=CSVOutputAdapter("errors.csv"),
schema_model=UserSchema,
)
pipe.start()
pipe.wait()
print(f"Finished! Processed {pipe.report.total_processed} items.")
Automatically split large files or manage multiple independent workflows:
from zoopipe import PipeManager, MultiProcessEngine
# Create your pipe as usual (Pipe is purely declarative)
pipe = Pipe(...)
# Automatically parallelize across 4 workers
# MultiProcessEngine() for local, RayEngine() for clusters
manager = PipeManager.parallelize_pipe(
pipe,
workers=4,
engine=MultiProcessEngine()
)
manager.start()
manager.wait()
📚 Documentation
Core Concepts
Hooks
Hooks are Python classes that allow you to intercept, transform, and enrich data at different stages of the pipeline.
📘 Read the full Hooks Guide to learn about lifecycle methods (setup, execute, teardown), state management, and advanced patterns like cursor pagination.
Quick Example
from zoopipe import BaseHook
class MyHook(BaseHook):
def execute(self, entries, store):
for entry in entries:
entry["raw_data"]["checked"] = True
return entries
Executors
Executors control how ZooPipe scales up within a single node using Rust-managed threads. They are the engine under the hood that drives high throughput.
📘 Read the full Executors Guide to understand the difference between SingleThreadExecutor (debug/ordered) and MultiThreadExecutor (high-throughput).
Input/Output Adapters
File Formats
- CSV Adapters - High-performance CSV reading and writing
- JSON Adapters - JSONL and JSON array format support
- Excel Adapters - Read and write Excel (.xlsx) files
- Parquet Adapters - Columnar storage for analytics and data lakes
- Arrow Adapters - Apache Arrow IPC format for high-throughput interoperability
Databases
- SQL Adapters - Read from and write to SQL databases with batch optimization
- SQL Pagination - High-performance cursor-style pagination for large tables
- DuckDB Adapters - Analytical database for OLAP workloads
Messaging Systems
- Kafka Adapters - High-throughput messaging
Advanced
- Python Generator Adapters - In-memory streaming and testing
- Cloud Storage (S3) - Read and write data from Amazon S3 and compatible services
- PipeManager - Run multiple pipes in parallel for distributed processing
- Ray Guide - Zero-config distributed execution on Ray clusters
🛠 Architecture
ZooPipe is designed as a thin Python wrapper around a powerful Rust core, featuring a two-tier parallel architecture:
- Orchestration Tier (Python Engines):
- Manage distribution across processes or nodes (e.g.,
MultiProcessEngine). - Handles data sharding, process lifecycle, and metrics aggregation.
- Manage distribution across processes or nodes (e.g.,
- Execution Tier (Rust BatchExecutors):
- Internal Throughput: High-speed processing within a single process.
- Adapters: Native CSV/JSON/SQL Readers and Writers.
- NativePipe: Orchestrates the loop, fetching chunks and routing result batches.
- Executors: Multi-threaded Rust strategies to bypass the GIL within a node.
📄 License
This project is licensed under the MIT License - see the LICENSE file for details.
Metadata
Release files for zoopipe 2026.1.20
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| zoopipe-2026.1.20.tar.gz | 219.1 kB | With provenance | Details for zoopipe-2026.1.20.tar.gz |
Built distributions (wheels)
Total release size: 149.9 MB
Release files / zoopipe-2026.1.20.tar.gz
| Download URL | zoopipe-2026.1.20.tar.gz |
|---|---|
| Size | 219.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
dd998184e8394657fead2bc1db01c6dba236ad7839a3b65ad42bf44ccb982bf8
|
|
BLAKE2b-256 checksum How to use checksums |
e9ddc4c1ea710943a279dcbb0e95eb9dfb2a19188c9c752221be89d0c423955d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 20, 2026.
Transparency logRelease files / zoopipe-2026.1.20-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl
| Download URL | zoopipe-2026.1.20-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 27.6 MB |
| Tags | Linux glibc 2.28+ x86-64 PyPy 3.11 PyPy 3.11 7.3 |
|
SHA-256 checksum How to use checksums |
cbc2d8b5cd819b376813901f7e24164329fd38f02657497d7959e7c96b711c09
|
|
BLAKE2b-256 checksum How to use checksums |
4827ec22a3e9735a3d4d049dfdaa7f594b25a936a8c7ea74fac6ff7c0e3a8a60
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 20, 2026.
Transparency logRelease files / zoopipe-2026.1.20-cp310-abi3-win_amd64.whl
| Download URL | zoopipe-2026.1.20-cp310-abi3-win_amd64.whl |
|---|---|
| Size | 21.1 MB |
| Tags | CPython 3.10 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
76d6a7b07726343acfa907f966199ee467554e0bcc447c024baae5bb5752b4e6
|
|
BLAKE2b-256 checksum How to use checksums |
4a688cbfb9fec883120b6e0d74808f656463c64931a7e7b06d9574a2263bba01
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 20, 2026.
Transparency logRelease files / zoopipe-2026.1.20-cp310-abi3-manylinux_2_28_x86_64.whl
| Download URL | zoopipe-2026.1.20-cp310-abi3-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 27.6 MB |
| Tags | CPython 3.10 Linux glibc 2.28+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
092d9a4e54d0afae7e7aa02f5e05f549d0e1bf8380ccc04a267dd7dabd2128b3
|
|
BLAKE2b-256 checksum How to use checksums |
4a5653116dbd5ba8a5810af8559a5ba79a8b3936e0eef8e69430b3c6782521b3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 20, 2026.
Transparency logRelease files / zoopipe-2026.1.20-cp310-abi3-manylinux_2_28_aarch64.whl
| Download URL | zoopipe-2026.1.20-cp310-abi3-manylinux_2_28_aarch64.whl |
|---|---|
| Size | 26.1 MB |
| Tags | CPython 3.10 Linux glibc 2.28+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
7092e7edefcaefe7247c20bc3f546675b81ca920882df9adf3a6f30ff19f7c70
|
|
BLAKE2b-256 checksum How to use checksums |
6023416fd045d035903444d4fcfb4e48ccde6fc3898d11cafccd0a1a925dd3b1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 20, 2026.
Transparency logRelease files / zoopipe-2026.1.20-cp310-abi3-macosx_11_0_arm64.whl
| Download URL | zoopipe-2026.1.20-cp310-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 23.0 MB |
| Tags | CPython 3.10 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
1effa14ec18492eccab07c0f7a74a3a3f0a29474be2300be9d46dec16d7b73d2
|
|
BLAKE2b-256 checksum How to use checksums |
f3fc9958a21ec054e3e6ff9d4e891da64337b5359f905c4be57e725ee901e2f3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 20, 2026.
Transparency logRelease files / zoopipe-2026.1.20-cp310-abi3-macosx_10_12_x86_64.whl
| Download URL | zoopipe-2026.1.20-cp310-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 24.3 MB |
| Tags | CPython 3.10 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
2940facec544ed8fcb0311c5ddc38250a5944f6b7f8604d42c4ebf3f0fa70698
|
|
BLAKE2b-256 checksum How to use checksums |
b20bf2e0989d3cb7193ffadd7814680a007473a60793a9c25467bbe39ee4b2ec
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 20, 2026.
Transparency logSecurity for zoopipe 2026.1.20
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 7 files in this release have been uploaded with provenance metadata.
Attestation for zoopipe-2026.1.20.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jan 20, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.