Valved

Agent-native data engineering, from natural language to production pipelines.

Valved takes a goal in plain English — "make stg_orders incremental", "ingest the Stripe charges API into a Snowflake table" — turns it into a reviewable plan, and, when you approve, materializes the working code and opens a pull request in your repo: dbt models, Python dlt sources, SQL, tests, schema docs. All the boring parts of building and maintaining a warehouse pipeline, executed by agents that read your conventions and respect your existing project. It's an AI-first, open-source control plane over dlt + dbt — self-hosted, Apache 2.0.

Status: v0.1.0 — the first release. The full loop (init → plan → build → run → deploy → schedule) is built and green across 3,000+ tests, with CLI / REST / MCP parity. Pre-1.0: it's usable and complete, with room to refine the API before committing to 1.0 stability.

Install

pip install valved

Requires Python 3.11+. Valved runs against your own Postgres (a bundled docker compose is scaffolded for you) and your own warehouse (Snowflake is first-class).

Quickstart

valved init --with-dbt              # scaffold a dlt + dbt project (+ bundled Postgres)
docker compose up -d                # start the state store
cp .env.example .env                # add ANTHROPIC_API_KEY + your Snowflake target

# intent → reviewable plan → working code
valved plan "ingest the Stripe charges API into raw_stripe, then a staging model stg_charges"
valved build <plan_id>              # writes the dlt source, the dbt model, and pipelines/<name>.toml
valved run <pipeline> --watch       # execute on demand; data lands in your warehouse
valved deploy <pipeline>            # open a reviewable PR with the built code (default handoff)

valved auto-loads a project-local .env on startup (existing shell vars win). Set VALVED_NO_DOTENV=1 to disable it if you manage env vars elsewhere (direnv, mise, 1Password CLI).

What it does

The core loop — every step is a valved verb, with a REST + MCP equivalent:

• plan → build — an AI harness turns intent into a reviewable Plan, then materializes it (dlt sources, dbt models, pipeline composition). It verifies by executing, not guessing. plan-and-build is the one-shot path.
• run / serve / worker — on-demand and scheduled execution over a durable job queue + worker pool (cron scheduler, crash-recovery reaper, graceful drain).
• deploy — promote built code to version control via a configurable handoff (files → commit → push → pr, default PR), including cross-repo linked PRs. Valved opens and orders the PRs; you merge.
• ask — a read-only investigative explorer: "where does net_revenue come from?", "what breaks if I change raw_stripe.charges?" — answered with citations by investigating dbt's manifest + dlt's schema + your code.
• Recovery — when a scheduled run fails, Valved diagnoses it (grounded in dlt's real exceptions) and proposes a reviewable fix — it never writes to prod autonomously.

Surfaces — the same functionality, four ways:

• CLI — the full valved command set (reference).
• REST API — FastAPI with full CLI parity, auth, streaming, webhooks (/api/openapi.json).
• MCP server — auto-generated from REST; drive Valved from Claude Desktop / Cursor / Claude Code (valved mcp-serve).
• Static HTML UI — run history + per-run logs (valved docs serve).

Documentation

• First run: with bundled Postgres · with external Postgres · installation
• Reference: CLI · config schema · glossary
• Design: specs/PRD.md (what/why) · specs/ARCHITECTURE.md (how) · specs/DELIVERY.md (build history)

Open source & governance

Valved is Apache 2.0 and feature-complete for single-team self-hosters — no API endpoints or MCP tools are gated behind any commercial offering. See OSS_COMMITMENT.md for what's permanently open, and governance for the model.

Contributing

Contributions welcome — DCO sign-off (git commit -s), dev setup, and PR expectations are in CONTRIBUTING.md. Report security issues privately (see SECURITY.md).

License

Apache 2.0. See LICENSE.