ZooPipe
ZooPipe is a lightweight, high-performance data processing framework for Python that combines the simplicity of Pydantic validation with the power of parallel processing.
Whether you're migrating data, cleaning CSVs, or processing streams, ZooPipe provides a structured way to handle validation, transformation, and error management without the complexity of big data frameworks.
✨ Key Features
- 🔍 Declarative Validation: Use Pydantic models to define and validate your data structures
- 🔌 Pluggable Architecture: Easily swap Input Adapters, Output Adapters, and Executors
- ⚡ Parallel Processing: Scale from single-threaded to distributed computing with
MultiprocessingExecutor,ThreadExecutor,DaskExecutorandRayExecutor - 🗜️ High-Performance Serialization: Uses msgpack and optional LZ4 compression for efficient inter-process communication
- 📊 Built-in Format Support: Direct support for CSV, JSON (array & JSONL), and Parquet files
- 🚨 Automated Error Handling: Dedicated error output adapter to capture records that fail validation
- 🪝 Hooks System: Transform and enrich data at various pipeline stages with built-in and custom hooks
- 🔄 Async Ready: Base adapters provided for async implementations
- 🛡️ Type Safe: Fully type-hinted for a better developer experience
🚀 Quick Start
Installation
uv add zoopipe
Or using pip:
pip install zoopipe
Simple Example
from pydantic import BaseModel, ConfigDict
from zoopipe import Pipe
from zoopipe.executor.sync_fifo import SyncFifoExecutor
from zoopipe.input_adapter.csv import CSVInputAdapter
from zoopipe.output_adapter.csv import CSVOutputAdapter
class UserSchema(BaseModel):
model_config = ConfigDict(extra="ignore")
name: str
last_name: str
age: int
pipe = Pipe(
input_adapter=CSVInputAdapter("users.csv"),
output_adapter=CSVOutputAdapter("processed_users.csv"),
error_output_adapter=CSVOutputAdapter("errors.csv"),
executor=SyncFifoExecutor(UserSchema),
)
# Start the pipewith context manager to ensure cleanup
with pipe:
report = pipe.start()
report.wait()
print(f"Finished! Processed {report.total_processed} items.")
📚 Documentation
Getting Started
- Installation & First Steps - Get up and running quickly
Core Concepts
- Executors - Learn about SyncFifoExecutor, MultiprocessingExecutor, ThreadExecutor, DaskExecutor and RayExecutor
- Adapters - Input and Output adapters for various data sources
- Examples - Practical examples for common use cases
🎯 Use Cases
ZooPipe excels at:
- Legacy Data Migrations: Moving data between heterogeneous databases with validation
- ETL Pipelines: Extract, Transform, Load workflows with error handling
- Data Cleaning: Processing manually generated files (Excel/CSV) with inconsistent formats
- Quality Filters: Acting as a validation layer before loading data into Data Lakes or ML models
- Batch Processing: Processing large datasets with parallel execution
🧩 Architecture
ZooPipe uses a decoupled architecture based on four components:
┌─────────────────┐ ┌──────────────┐ ┌─────────────────┐
│ Input Adapter │─────▶│ Executor │─────▶│ Output Adapter │
│ (CSV, JSON, │ │ (Validation │ │ (CSV, JSON, │
│ Parquet, DB, │ │ & Transform)│ │ Parquet, DB, │
│ API, etc) │ │ │ │ API, etc) │
└─────────────────┘ └──────────────┘ └─────────────────┘
│
│ (errors)
▼
┌──────────────┐
│ Error Output │
│ Adapter │
└──────────────┘
- Input Adapter: Reads data from sources (CSV, JSON, Parquet, SQL, API)
- Executor: Validates with Pydantic and processes data (sequential or parallel)
- Output Adapter: Persists validated data
- Error Output Adapter: Captures failed validations (Dead Letter Queue)
Learn more in the Architecture RFC.
🔧 Executors
ZooPipe provides three execution strategies:
| Executor | Best For | Parallelism |
|---|---|---|
SyncFifoExecutor |
Small datasets, debugging | Single-threaded |
MultiprocessingExecutor |
Large datasets on single machine | Multi-process (CPU cores) |
ThreadExecutor |
IO-bound tasks (network/DB) | Multi-thread |
DaskExecutor |
ETL pipelines, Dask users | Dask cluster |
RayExecutor |
Massive datasets, distributed | Ray cluster |
See the Executors documentation for detailed information.
🛠 Development
Setup
git clone https://github.com/albertobadia/zoopipe.git
cd zoopipe
uv sync
Running Tests
uv run pytest -v
Linting
./lint.sh
📄 License
This project is licensed under the MIT License - see the LICENSE file for details.
🤝 Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
📧 Contact
Alberto Daniel Badia
Email: alberto_badia@enlacepatagonia.com
GitHub: @albertobadia
Metadata
Release files for zoopipe 2026.1.9
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| zoopipe-2026.1.9.tar.gz | 30.9 kB | Without provenance | Details for zoopipe-2026.1.9.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| zoopipe-2026.1.9-py3-none-any.whl | Python 3 | none | any | Details for zoopipe-2026.1.9-py3-none-any.whl |
Total release size: 62.6 kB
Release files / zoopipe-2026.1.9.tar.gz
| Download URL | zoopipe-2026.1.9.tar.gz |
|---|---|
| Size | 30.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ee8af7772f4fac9d3187c1e1d579c79d3c615958fc784c747ba40c18c9392a4e
|
|
BLAKE2b-256 checksum How to use checksums |
8327dc507156f393b282fad1ebe3d9467afe676c1b040b715ac4632e6352fd3f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.9.22 {"installer":{"name":"uv","version":"0.9.22","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Arch Linux ARM","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / zoopipe-2026.1.9-py3-none-any.whl
| Download URL | zoopipe-2026.1.9-py3-none-any.whl |
|---|---|
| Size | 31.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f51bf0636b67323c70eeb660fbd85c21fc225d4d559470a9163ebb38dc39d68d
|
|
BLAKE2b-256 checksum How to use checksums |
bff98f5c194398e91e488e54cc78eb326ceeabb1050e04e95a151411b27a23be
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.9.22 {"installer":{"name":"uv","version":"0.9.22","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Arch Linux ARM","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for zoopipe 2026.1.9
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.