Skip to main content

ZooPipe Logo

ZooPipe is a lean, ultra-high-performance data processing engine for Python. It leverages a 100% Rust core to handle I/O and orchestration, while keeping the flexibility of Python for schema validation (via Pydantic) and custom data enrichment (via Hooks).

PyPI Downloads CI ReadTheDocs


Read the docs for more information.

✨ Key Features

  • 🚀 100% Native Rust Engine: The core execution loop, including CSV and JSON parsing/writing, is implemented in Rust for maximum throughput.
  • 🔍 Declarative Validation: Use Pydantic models to define and validate your data structures naturally.
  • 🪝 Python Hooks: Transform and enrich data at any stage using standard Python functions or classes.
  • 🚨 Automated Error Routing: Native support for routing failed records to a dedicated error output.
  • 📊 Multiple Format Support: Optimized readers/writers for CSV, JSONL, and SQL databases.
  • 🔧 Two-Tier Parallelism: Orchestrate across processes or clusters with Engines (Local, Ray), and scale throughput at the node level with Rust Executors.
  • ☁️ Cloud Native: Native S3 support and zero-config distributed execution on Ray clusters.

⚡ Performance & Benchmarks

Why ZooPipe? Because vectorization isn't always the answer.

Tools like Pandas and Polars are incredible for analytical workloads (groupby, sum, joins) where operations can be vectorized in C/Rust. However, real-world Data Engineering often involves "chaotic ETL": messy custom rules, API calls per row, hashing, conditional cleanup, and complex normalization that forcedly drop down to Python loops.

In these "Heavy ETL" scenarios, ZooPipe outperforms Vectorized DataFrames by 3x-8x.

Benchmark Chart

Key Takeaway: ZooPipe's "Python-First Architecture" with parallel streaming (PipeManager) avoids the serialization overhead that cripples Polars/Pandas when using Python UDFs (map_elements/apply), and uses 97% less RAM.

⚖️ Is this unfair to Pandas/Polars?

Yes and No.

  • Unfair: If your workload is purely analytical (e.g., GROUP BY, SUM, JOIN), Polars and Pandas will likely destroy ZooPipe because they can use vectorized C/Rust operations on whole columns at once.
  • Fair: In real-world Data Engineering, many pipelines are "chaotic". They require custom hashing, API calls per row, conditional normalization, or complex Pydantic validation. In these "Python-UDF heavy" scenarios, vectorization breaks down, and ZooPipe shines by orchestrating parallel Python execution efficiently without the DataFrame overhead.

❓ When to use what?

Use ZooPipe When... Use Pandas / Polars When...
🏗️ You have complex, custom Python logic per row (hash, clean, validate). 🧮 You are doing aggregations (SUM, AVG) or Relational Algebra (JOIN, GROUP BY).
🔄 You are processing streaming data or files larger than RAM. 💾 Your dataset fits comfortably in RAM (or use LazyFrames).
🛡️ You need strict schema validation (Pydantic) and error handling. 🔬 You are doing data exploration or statistical analysis.
🚀 You want to mix Rust I/O performance with Python flexibility. ⚡ Your entire pipeline can be expressed in vectorized expressions.

🚀 Quick Start

Installation

pip install zoopipe

Or using uv:

uv add zoopipe

Or from source (uv recommended):

uv build
uv run maturin develop --release

Simple Example

from pydantic import BaseModel, ConfigDict
from zoopipe import CSVInputAdapter, CSVOutputAdapter, Pipe


class UserSchema(BaseModel):
    model_config = ConfigDict(extra="ignore")
    user_id: str
    username: str
    email: str


pipe = Pipe(
    input_adapter=CSVInputAdapter("users.csv"),
    output_adapter=CSVOutputAdapter("processed_users.csv"),
    error_output_adapter=CSVOutputAdapter("errors.csv"),
    schema_model=UserSchema,
)

pipe.start()
pipe.wait()


print(f"Finished! Processed {pipe.report.total_processed} items.")

Automatically split large files or manage multiple independent workflows:

from zoopipe import PipeManager, MultiProcessEngine

# Create your pipe as usual (Pipe is purely declarative)
pipe = Pipe(...)

# Automatically parallelize across 4 workers
# MultiProcessEngine() for local, RayEngine() for clusters
manager = PipeManager.parallelize_pipe(
    pipe, 
    workers=4, 
    engine=MultiProcessEngine() 
)
manager.start()
manager.wait()

📚 Documentation

Core Concepts

Hooks

Hooks are Python classes that allow you to intercept, transform, and enrich data at different stages of the pipeline.

📘 Read the full Hooks Guide to learn about lifecycle methods (setup, execute, teardown), state management, and advanced patterns like cursor pagination.

Quick Example

from zoopipe import BaseHook

class MyHook(BaseHook):
    def execute(self, entries, store):
        for entry in entries:
            entry["raw_data"]["checked"] = True
        return entries

Executors

Executors control how ZooPipe scales up within a single node using Rust-managed threads. They are the engine under the hood that drives high throughput.

📘 Read the full Executors Guide to understand the difference between SingleThreadExecutor (debug/ordered) and MultiThreadExecutor (high-throughput).

Input/Output Adapters

File Formats

Databases

  • SQL Adapters - Read from and write to SQL databases with batch optimization
  • SQL Pagination - High-performance cursor-style pagination for large tables
  • DuckDB Adapters - Analytical database for OLAP workloads

Messaging Systems

Advanced


🛠 Architecture

ZooPipe is designed as a thin Python wrapper around a powerful Rust core, featuring a two-tier parallel architecture:

  1. Orchestration Tier (Python Engines):
    • Manage distribution across processes or nodes (e.g., MultiProcessEngine).
    • Handles data sharding, process lifecycle, and metrics aggregation.
  2. Execution Tier (Rust BatchExecutors):
    • Internal Throughput: High-speed processing within a single process.
    • Adapters: Native CSV/JSON/SQL Readers and Writers.
    • NativePipe: Orchestrates the loop, fetching chunks and routing result batches.
    • Executors: Multi-threaded Rust strategies to bypass the GIL within a node.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

Metadata

Release files for zoopipe 2026.1.20

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for zoopipe 2026.1.20
File Size Uploaded Provenance Actions
zoopipe-2026.1.20.tar.gz 219.1 kB With provenance Details for zoopipe-2026.1.20.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for zoopipe 2026.1.20
File Actions
zoopipe-2026.1.20-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl PyPy 3.11 PyPy 3.11 7.3 Linux glibc 2.28+ x86-64 Details for zoopipe-2026.1.20-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl
zoopipe-2026.1.20-cp310-abi3-win_amd64.whl CPython 3.10 abi3 Windows x86-64 Details for zoopipe-2026.1.20-cp310-abi3-win_amd64.whl
zoopipe-2026.1.20-cp310-abi3-manylinux_2_28_x86_64.whl CPython 3.10 abi3 Linux glibc 2.28+ x86-64 Details for zoopipe-2026.1.20-cp310-abi3-manylinux_2_28_x86_64.whl
zoopipe-2026.1.20-cp310-abi3-manylinux_2_28_aarch64.whl CPython 3.10 abi3 Linux glibc 2.28+ ARM64 Details for zoopipe-2026.1.20-cp310-abi3-manylinux_2_28_aarch64.whl
zoopipe-2026.1.20-cp310-abi3-macosx_11_0_arm64.whl CPython 3.10 abi3 macOS 11.0+ ARM64 Details for zoopipe-2026.1.20-cp310-abi3-macosx_11_0_arm64.whl
zoopipe-2026.1.20-cp310-abi3-macosx_10_12_x86_64.whl CPython 3.10 abi3 macOS 10.12+ x86-64 Details for zoopipe-2026.1.20-cp310-abi3-macosx_10_12_x86_64.whl

Total release size: 149.9 MB

Release files / zoopipe-2026.1.20.tar.gz

Download URL zoopipe-2026.1.20.tar.gz
Size 219.1 kB
Tags Source
SHA-256 checksum
How to use checksums
dd998184e8394657fead2bc1db01c6dba236ad7839a3b65ad42bf44ccb982bf8
BLAKE2b-256 checksum
How to use checksums
e9ddc4c1ea710943a279dcbb0e95eb9dfb2a19188c9c752221be89d0c423955d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jan 20, 2026.

Transparency log

Release files / zoopipe-2026.1.20-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl

Download URL zoopipe-2026.1.20-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl
Size 27.6 MB
Tags Linux glibc 2.28+ x86-64 PyPy 3.11 PyPy 3.11 7.3
SHA-256 checksum
How to use checksums
cbc2d8b5cd819b376813901f7e24164329fd38f02657497d7959e7c96b711c09
BLAKE2b-256 checksum
How to use checksums
4827ec22a3e9735a3d4d049dfdaa7f594b25a936a8c7ea74fac6ff7c0e3a8a60
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jan 20, 2026.

Transparency log

Release files / zoopipe-2026.1.20-cp310-abi3-win_amd64.whl

Download URL zoopipe-2026.1.20-cp310-abi3-win_amd64.whl
Size 21.1 MB
Tags CPython 3.10 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
76d6a7b07726343acfa907f966199ee467554e0bcc447c024baae5bb5752b4e6
BLAKE2b-256 checksum
How to use checksums
4a688cbfb9fec883120b6e0d74808f656463c64931a7e7b06d9574a2263bba01
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jan 20, 2026.

Transparency log

Release files / zoopipe-2026.1.20-cp310-abi3-manylinux_2_28_x86_64.whl

Download URL zoopipe-2026.1.20-cp310-abi3-manylinux_2_28_x86_64.whl
Size 27.6 MB
Tags CPython 3.10 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum
How to use checksums
092d9a4e54d0afae7e7aa02f5e05f549d0e1bf8380ccc04a267dd7dabd2128b3
BLAKE2b-256 checksum
How to use checksums
4a5653116dbd5ba8a5810af8559a5ba79a8b3936e0eef8e69430b3c6782521b3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jan 20, 2026.

Transparency log

Release files / zoopipe-2026.1.20-cp310-abi3-manylinux_2_28_aarch64.whl

Download URL zoopipe-2026.1.20-cp310-abi3-manylinux_2_28_aarch64.whl
Size 26.1 MB
Tags CPython 3.10 Linux glibc 2.28+ ARM64 abi3
SHA-256 checksum
How to use checksums
7092e7edefcaefe7247c20bc3f546675b81ca920882df9adf3a6f30ff19f7c70
BLAKE2b-256 checksum
How to use checksums
6023416fd045d035903444d4fcfb4e48ccde6fc3898d11cafccd0a1a925dd3b1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jan 20, 2026.

Transparency log

Release files / zoopipe-2026.1.20-cp310-abi3-macosx_11_0_arm64.whl

Download URL zoopipe-2026.1.20-cp310-abi3-macosx_11_0_arm64.whl
Size 23.0 MB
Tags CPython 3.10 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
1effa14ec18492eccab07c0f7a74a3a3f0a29474be2300be9d46dec16d7b73d2
BLAKE2b-256 checksum
How to use checksums
f3fc9958a21ec054e3e6ff9d4e891da64337b5359f905c4be57e725ee901e2f3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jan 20, 2026.

Transparency log

Release files / zoopipe-2026.1.20-cp310-abi3-macosx_10_12_x86_64.whl

Download URL zoopipe-2026.1.20-cp310-abi3-macosx_10_12_x86_64.whl
Size 24.3 MB
Tags CPython 3.10 abi3 macOS 10.12+ x86-64
SHA-256 checksum
How to use checksums
2940facec544ed8fcb0311c5ddc38250a5944f6b7f8604d42c4ebf3f0fa70698
BLAKE2b-256 checksum
How to use checksums
b20bf2e0989d3cb7193ffadd7814680a007473a60793a9c25467bbe39ee4b2ec
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jan 20, 2026.

Transparency log

Security for zoopipe 2026.1.20

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 7 files in this release have been uploaded with provenance metadata.

Select from 7 files

Attestation for zoopipe-2026.1.20.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jan 20, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page