Zero to Pipeline

Self-configuring data ingestion — connect any API without writing connectors.

Zero to Pipeline is a Python framework that turns any REST or GraphQL API into a data source in a single command. No connector classes, no YAML schemas, no pagination boilerplate. The framework uses an LLM (your own API key) to discover auth type, endpoints, and pagination style from a provider name — then extracts data with self-healing, checkpointed syncs.

Tutorial: Build your first pipeline

Connect to an API, store credentials, and extract data in under five minutes.

Prerequisites

• Python 3.10+
• An API key for OpenAI or Anthropic (or a local Ollama instance)

Step 1 — Install

pip install zero-to-pipeline

Step 2 — Store your LLM API key

pipeline auth set openai --token sk-proj-...

Step 3 — Add a data source

pipeline source add mlflow

Step 4 — Test the connection

pipeline source test mlflow

You should see: Connection to mlflow successful!

Step 5 — Extract data

pipeline sync run mlflow

The first run fetches all records. The second run resumes from the last cursor — only new records.

What you learned

You connected to an API, stored credentials securely, and extracted paginated data — without writing a single line of connector code.

How-to guides

Explanation: How it works

Traditional ETL tools require you to write a connector for every API. Zero to Pipeline inverts the problem: you say what to connect, and the framework figures out how.

pipeline source add <provider>
        │
        ▼
┌─────────────────────┐
│  1. Provider         │  Known presets (MLflow, GitHub, Airflow, etc.)
│     Registry         │  give instant base URL, auth, pagination
└────────┬────────────┘
         │ enriches
         ▼
┌─────────────────────┐
│  2. LLM Discovery    │  Your LLM fills gaps: endpoints, rate limits,
│                      │  API quirks. Unknown providers get full config.
└────────┬────────────┘
         │ validates
         ▼
┌─────────────────────┐
│  3. HTTP Probing     │  Checks reachability, detects pagination
│                      │  from response headers and body shape
└────────┬────────────┘
         │ connects
         ▼
┌─────────────────────┐
│  4. Self-Healing     │  Rotates auth formats on 401/403.
│     Connector        │  Resumes from last cursor after healing.
└────────┬────────────┘
         │ extracts
         ▼
┌─────────────────────┐
│  5. Checkpointed     │  Saves cursor every 100 records.
│     Extraction       │  Next run: resumes, doesn't replay.
└─────────────────────┘

See docs/explanation.md for the full architecture deep dive.

Reference

Supported providers

Known presets (demo accelerators — any API name works without them):

Configuration

All settings support .env files. See docs/reference.md for the complete list.

CLI commands

Project structure

src/data_pipeline/
├── cli/                 # CLI commands (package, not a single file)
├── connectors/          # API connectivity layer
├── auth/                # Credential storage
├── orchestrator/        # Pipeline execution
├── extractors/          # Orchestration-aware extraction
├── loaders/             # JSONL output (extensible)
├── schemas/             # Pydantic models
├── sources/             # Source state persistence
├── mcp/                 # Model Context Protocol server
└── observability/       # Structured logging + metrics

Development

git clone https://github.com/Lanrey/zero-to-pipeline.git
cd zero-to-pipeline
uv sync --extra dev

pytest tests/ -v            # 79 tests
ruff check src/data_pipeline/   # zero lint errors
mypy src/data_pipeline/         # type check

See CONTRIBUTING.md for the full style guide and pre-submit checklist.

License

MIT