Skip to main content

ZooPipe

ZooPipe is a lightweight, high-performance data processing framework for Python that combines the simplicity of Pydantic validation with the power of parallel processing.

Whether you're migrating data, cleaning CSVs, or processing streams, ZooPipe provides a structured way to handle validation, transformation, and error management without the complexity of big data frameworks.


✨ Key Features

  • 🔍 Declarative Validation: Use Pydantic models to define and validate your data structures
  • 🔌 Pluggable Architecture: Easily swap Input Adapters, Output Adapters, and Executors
  • ⚡ Parallel Processing: Scale from single-threaded to distributed computing with MultiprocessingExecutor, ThreadExecutor, DaskExecutor and RayExecutor
  • 🗜️ High-Performance Serialization: Uses msgpack and optional LZ4 compression for efficient inter-process communication
  • 📊 Built-in Format Support: Direct support for CSV, JSON (array & JSONL), and Parquet files
  • 🚨 Automated Error Handling: Dedicated error output adapter to capture records that fail validation
  • 🪝 Hooks System: Transform and enrich data at various pipeline stages with built-in and custom hooks
  • 🔄 Async Ready: Base adapters provided for async implementations
  • 🛡️ Type Safe: Fully type-hinted for a better developer experience

🚀 Quick Start

Installation

uv add zoopipe

Or using pip:

pip install zoopipe

Simple Example

from pydantic import BaseModel, ConfigDict
from zoopipe import Pipe
from zoopipe.executor.sync_fifo import SyncFifoExecutor
from zoopipe.input_adapter.csv import CSVInputAdapter
from zoopipe.output_adapter.csv import CSVOutputAdapter

class UserSchema(BaseModel):
    model_config = ConfigDict(extra="ignore")
    name: str
    last_name: str
    age: int

pipe = Pipe(
    input_adapter=CSVInputAdapter("users.csv"),
    output_adapter=CSVOutputAdapter("processed_users.csv"),
    error_output_adapter=CSVOutputAdapter("errors.csv"),
    executor=SyncFifoExecutor(UserSchema),
)

# Start the pipewith context manager to ensure cleanup
with pipe:
    report = pipe.start()
    report.wait()

print(f"Finished! Processed {report.total_processed} items.")

📚 Documentation

Getting Started

Core Concepts

  • Executors - Learn about SyncFifoExecutor, MultiprocessingExecutor, ThreadExecutor, DaskExecutor and RayExecutor
  • Adapters - Input and Output adapters for various data sources
  • Examples - Practical examples for common use cases

🎯 Use Cases

ZooPipe excels at:

  • Legacy Data Migrations: Moving data between heterogeneous databases with validation
  • ETL Pipelines: Extract, Transform, Load workflows with error handling
  • Data Cleaning: Processing manually generated files (Excel/CSV) with inconsistent formats
  • Quality Filters: Acting as a validation layer before loading data into Data Lakes or ML models
  • Batch Processing: Processing large datasets with parallel execution

🧩 Architecture

ZooPipe uses a decoupled architecture based on four components:

┌─────────────────┐      ┌──────────────┐      ┌─────────────────┐
│  Input Adapter  │─────▶│   Executor   │─────▶│ Output Adapter  │
│ (CSV, JSON,     │      │ (Validation  │      │ (CSV, JSON,     │
│  Parquet, DB,   │      │  & Transform)│      │  Parquet, DB,   │
│    API, etc)    │      │              │      │   API, etc)     │
└─────────────────┘      └──────────────┘      └─────────────────┘
                                │
                                │ (errors)
                                ▼
                         ┌──────────────┐
                         │ Error Output │
                         │   Adapter    │
                         └──────────────┘
  • Input Adapter: Reads data from sources (CSV, JSON, Parquet, SQL, API)
  • Executor: Validates with Pydantic and processes data (sequential or parallel)
  • Output Adapter: Persists validated data
  • Error Output Adapter: Captures failed validations (Dead Letter Queue)

Learn more in the Architecture RFC.


🔧 Executors

ZooPipe provides three execution strategies:

Executor Best For Parallelism
SyncFifoExecutor Small datasets, debugging Single-threaded
MultiprocessingExecutor Large datasets on single machine Multi-process (CPU cores)
ThreadExecutor IO-bound tasks (network/DB) Multi-thread
DaskExecutor ETL pipelines, Dask users Dask cluster
RayExecutor Massive datasets, distributed Ray cluster

See the Executors documentation for detailed information.


🛠 Development

Setup

git clone https://github.com/albertobadia/zoopipe.git
cd zoopipe
uv sync

Running Tests

uv run pytest -v

Linting

./lint.sh

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.


🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.


📧 Contact

Alberto Daniel Badia
Email: alberto_badia@enlacepatagonia.com
GitHub: @albertobadia

Metadata

Release files for zoopipe 2026.1.9

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for zoopipe 2026.1.9
File Size Uploaded Provenance Actions
zoopipe-2026.1.9.tar.gz 30.9 kB Without provenance Details for zoopipe-2026.1.9.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for zoopipe 2026.1.9
File Interpreter ABI Platform Actions
zoopipe-2026.1.9-py3-none-any.whl Python 3 none any Details for zoopipe-2026.1.9-py3-none-any.whl

Total release size: 62.6 kB

Release files / zoopipe-2026.1.9.tar.gz

Download URL zoopipe-2026.1.9.tar.gz
Size 30.9 kB
Tags Source
SHA-256 checksum
How to use checksums
ee8af7772f4fac9d3187c1e1d579c79d3c615958fc784c747ba40c18c9392a4e
BLAKE2b-256 checksum
How to use checksums
8327dc507156f393b282fad1ebe3d9467afe676c1b040b715ac4632e6352fd3f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.9.22 {"installer":{"name":"uv","version":"0.9.22","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Arch Linux ARM","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / zoopipe-2026.1.9-py3-none-any.whl

Download URL zoopipe-2026.1.9-py3-none-any.whl
Size 31.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f51bf0636b67323c70eeb660fbd85c21fc225d4d559470a9163ebb38dc39d68d
BLAKE2b-256 checksum
How to use checksums
bff98f5c194398e91e488e54cc78eb326ceeabb1050e04e95a151411b27a23be
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.9.22 {"installer":{"name":"uv","version":"0.9.22","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Arch Linux ARM","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for zoopipe 2026.1.9

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page