dpone 0.1.0

Declarative ETL framework for YAML-driven data pipelines

dpone

dpone is a Python ETL framework for declarative, YAML-driven data pipelines. It helps data teams describe sources, sinks, load strategies, dependencies, conventions, and operational checks as reusable configuration instead of one-off scripts.

The public package name, import name, GitHub repository name, and CLI name are all intentionally short: dpone.

Repository: https://github.com/PaulKov/dpone

What dpone gives you

• YAML manifests for single-process and batch ETL definitions.
• Built-in DAG/dependency inspection for pipeline debugging.
• Runtime abstractions for sources, sinks, connectors, state, reconciliation, and safe SQL logging.
• Optional integrations for PostgreSQL, ClickHouse, BigQuery/GCS, pandas, Google Ads, and HashiCorp Vault.
• A CLI designed for self-service validation, rendering, explainability, and documentation checks.
• Compatibility shims for older import paths while the canonical package layout continues to stabilize.

Installation

Install the core package from PyPI:

pip install dpone

Install common extras for local ETL development:

pip install "dpone[postgres,clickhouse,gcp,pandas,vault]"

Install everything currently published by the project:

pip install "dpone[full]"

With uv:

uv add dpone
uv add "dpone[full]"

Optional extras

Extra	Purpose
postgres	PostgreSQL connectivity via psycopg
clickhouse	ClickHouse connectivity
gcp	Google BigQuery and Google Cloud Storage support
pandas	DataFrame-based extract/load helpers
vault	HashiCorp Vault integration via public vault-kv-client
google_ads	Google Ads API support
full	All public extras above

Vault support uses vault-kv-client, published on PyPI as vault-kv-client. New code should use vault_kv_client; the historical vault_client import path remains supported by that package as a compatibility layer.

Quick start

Create a batch manifest, for example examples/batch/landing_postgres_to_bq.batch.yaml:

# yaml-language-server: $schema=../../src/dpone/schema/etl-batch-manifest.schema.json
kind: dpone.batch.v1
convention: landing_raw_v1
registry: ../registry/sources.yaml

vars:
  src_system: demo_source
  src_database: demo_db
  owner_team: data-platform
  owner_contact: [e-mail]
  sla: daily

defaults:
  source:
    type: postgres
    connection_type: vault
    connection_id: postgres-demo
    vault_path: postgres/demo-source
    options:
      batch_size: 100000
      export_format: csv

  sink:
    type: bigquery
    connection_type: vault
    connection_id: bigquery-demo
    vault_path: gcp/demo-project-prod/bq/service-account
    staging:
      schema: stg
    strategy:
      mode: full_refresh
      overwrite_type: exchange

schemas:
  public:
    tables:
      - core_city

Validate and render it:

dpone manifest validate examples/batch/landing_postgres_to_bq.batch.yaml \
  --profile landing_raw_v1 \
  --registry examples/registry/sources.yaml

dpone manifest render examples/batch/landing_postgres_to_bq.batch.yaml \
  --selector public.core_city \
  --registry examples/registry/sources.yaml

Inspect pipeline dependencies:

dpone dag report examples/batch/landing_postgres_to_bq.batch.yaml \
  --base-path . \
  --format json \
  --preset ci \
  --registry examples/registry/sources.yaml

CLI overview

dpone --help
dpone manifest --help
dpone dag --help
dpone docs --help

Common commands:

dpone manifest list examples/batch/landing_postgres_to_bq.batch.yaml
dpone manifest validate examples/batch/landing_postgres_to_bq.batch.yaml --recursive
dpone manifest render examples/batch/landing_postgres_to_bq.batch.yaml --selector public.core_city
dpone manifest explain examples/batch/landing_postgres_to_bq.batch.yaml --selector public.core_city --why sink.table.schema
dpone dag list-edges examples/batch/landing_postgres_to_bq.batch.yaml --with-groups --with-refs
dpone dag explain-node examples/batch/landing_postgres_to_bq.batch.yaml --task public.core_city
dpone dag report examples/batch/landing_postgres_to_bq.batch.yaml --preset ci --format md

Repository layout

src/dpone/      Python package source code
docs/           User and developer documentation
examples/       Public example manifests and registries
tests/          Unit and integration tests
tools/          Local smoke and release helper scripts

Canonical imports live under:

• dpone.manifest.*
• dpone.dag.*
• dpone.runtime.*
• dpone.contracts.*
• dpone.ports.*
• dpone.adapters.*

Legacy paths such as dpone.core.*, dpone.lib.*, dpone.source.*, and dpone.sink.* are compatibility shims. Prefer canonical imports for new code.

Local development

uv sync --all-extras
uv run ruff check .
uv run ruff format --check .
uv run mypy --config-file mypy.ini
uv run pytest -m "not integration_live"

Build package artifacts:

uv build

Run the package smoke script from an installed environment:

python tools/package_smoke.py --project-root . --dpone-cmd dpone

CI and releases

The OSS repository uses GitHub Actions as the primary automation path:

• .github/workflows/ci.yml runs linting, formatting checks, type checks, tests, and package build.
• .github/workflows/release.yml builds and publishes tagged releases to PyPI.

Release tags use the format vX.Y.Z, for example:

git tag -a v0.1.0 -m "Release v0.1.0"
git push origin v0.1.0

Prefer PyPI Trusted Publishing for releases. Token-based publishing should only be used as a fallback with short-lived, scoped tokens.

Security

Never commit API tokens, PyPI tokens, GitHub tokens, Vault credentials, service-account JSON, or live vendor credentials. If a secret is ever pasted into an issue, chat, commit, or CI log, revoke it before publishing or pushing public history.

See SECURITY.md for the vulnerability reporting process.

Documentation

Start with docs/README.md. Key pages:

• Architecture
• CLI reference
• CLI examples
• Manifest conventions
• DAG debugging
• Testing
• Release

License

dpone is licensed under the Apache License 2.0. See LICENSE.