dpone

dpone is a Python ETL framework for declarative, YAML-driven data pipelines. It helps data teams describe sources, sinks, load strategies, dependencies, conventions, and operational checks as reusable configuration instead of one-off scripts.

The public package name, import name, GitHub repository name, and CLI name are all intentionally short: dpone.

Repository: https://github.com/PaulKov/dpone

Public install:

python3 -m pip install dpone
dpone --help

First Airflow DAG

Create and verify a provider-loadable DAG without writing Airflow Python or configuring credentials:

dpone init project --airflow
dpone init pipeline orders_daily --recipe mssql-to-clickhouse-incremental
dpone check orders_daily
dpone airflow preview orders_daily
dpone test orders_daily

The five commands are the complete offline beginner journey. A live, bounded sample is a separate platform-gated action. See First Airflow DAG.

Documentation map

Start here if you are evaluating or operating dpone:

What dpone gives you

YAML manifests for single-process and batch ETL definitions.
Built-in DAG/dependency inspection for pipeline debugging.
Runtime abstractions for sources, sinks, connectors, state, reconciliation, and safe SQL logging.
Optional integrations for PostgreSQL, MSSQL/SQL Server, ClickHouse, BigQuery/GCS, Kafka, pandas, Google Ads, and HashiCorp Vault.
A CLI designed for self-service validation, rendering, explainability, and documentation checks.
Compatibility shims for older import paths while the canonical package layout continues to stabilize.

Installation

Install the core package from PyPI:

pip install dpone
dpone --version
dpone -v

Install common extras for local ETL development:

pip install "dpone[postgres,mssql,clickhouse,kafka,gcp,s3,azure,pandas,vault]"

Install everything currently published by the project:

pip install "dpone[full]"

With uv:

uv add dpone
uv add "dpone[full]"

Install the Kubernetes SDK only in the restricted platform environment that runs Airflow Connection Secret GC or runtime-Pod retention:

pip install "dpone[kubernetes]"

Optional extras

Extra	Purpose
`postgres`	PostgreSQL connectivity via `psycopg`
`mssql`	Microsoft SQL Server connectivity via `pyodbc`; production bulk paths use external ODBC Driver 18 and `bcp`
`clickhouse`	ClickHouse connectivity
`gcp`	Google BigQuery and Google Cloud Storage support
`s3`	AWS S3 object storage staging support via `boto3`
`azure`	Azure Blob Storage plus workload identity via `azure-storage-blob` and `azure-identity`
`object_storage`	S3, GCS, and Azure Blob staging, including Azure workload identity
`kafka`	Kafka batch source/sink support via `confluent-kafka`, Schema Registry codecs, Avro, JSON Schema, and Protobuf helpers
`pandas`	DataFrame-based extract/load helpers
`vault`	HashiCorp Vault integration via public `vault-kv-client`
`kubernetes`	Metadata-only, namespace-scoped Airflow Connection Secret GC and runtime-Pod retention for platform operators
`google_ads`	Google Ads API support
`full`	All public extras above

Vault support uses vault-kv-client, published on PyPI as vault-kv-client. dpone imports the canonical vault_kv_client module only; legacy Vault import paths are not supported by dpone.

Credentials can come from environment variables, Airflow Connections, HashiCorp Vault-compatible KV, or inline params for smoke tests. See Connections and credentials for copy-paste examples for Postgres, MSSQL, ClickHouse, BigQuery, Kafka, and REST API.

Quick start

Create a batch manifest, for example examples/batch/landing_postgres_to_bq.batch.yaml:

# yaml-language-server: $schema=../../src/dpone/schema/etl-batch-manifest.schema.json
kind: dpone.batch.v1
convention: landing_raw_v1
registry: ../registry/sources.yaml

vars:
  src_system: demo_source
  src_database: demo_db
  owner_team: data-platform
  owner_contact: data-platform@example.com
  sla: daily

defaults:
  source:
    type: postgres
    connection_type: vault
    connection_id: postgres-demo
    vault_path: postgres/demo-source
    options:
      batch_size: 100000
      export_format: csv

  sink:
    type: bigquery
    connection_type: vault
    connection_id: bigquery-demo
    vault_path: gcp/demo-project-prod/bq/service-account
    staging:
      schema: stg
    strategy:
      mode: full_refresh
      overwrite_type: exchange

schemas:
  public:
    tables:
      - core_city

Validate and render it:

dpone manifest validate examples/batch/landing_postgres_to_bq.batch.yaml \
  --profile landing_raw_v1 \
  --registry examples/registry/sources.yaml

dpone manifest render examples/batch/landing_postgres_to_bq.batch.yaml \
  --selector public.core_city \
  --registry examples/registry/sources.yaml

Inspect pipeline dependencies:

dpone dag report examples/batch/landing_postgres_to_bq.batch.yaml \
  --base-path . \
  --format json \
  --preset ci \
  --registry examples/registry/sources.yaml

CLI overview

dpone --help
dpone manifest --help
dpone dag --help
dpone docs --help

Common commands:

dpone manifest list examples/batch/landing_postgres_to_bq.batch.yaml
dpone manifest validate examples/batch/landing_postgres_to_bq.batch.yaml --recursive
dpone manifest render examples/batch/landing_postgres_to_bq.batch.yaml --selector public.core_city
dpone manifest explain examples/batch/landing_postgres_to_bq.batch.yaml --selector public.core_city --why sink.table.schema
dpone dag list-edges examples/batch/landing_postgres_to_bq.batch.yaml --with-groups --with-refs
dpone dag explain-node examples/batch/landing_postgres_to_bq.batch.yaml --task public.core_city
dpone dag report examples/batch/landing_postgres_to_bq.batch.yaml --preset ci --format md

Repository layout

src/dpone/      Python package source code
docs/           User and developer documentation
examples/       Public example manifests and registries
tests/          Unit and integration tests
tools/          Local smoke and release helper scripts

Canonical imports live under:

dpone.manifest.*
dpone.dag.*
dpone.runtime.*
dpone.contracts.*
dpone.ports.*
dpone.adapters.*

Legacy paths such as dpone.core.*, dpone.lib.*, dpone.source.*, and dpone.sink.* are compatibility shims. Prefer canonical imports for new code.

Local development

uv sync --all-extras
uv run ruff check .
uv run ruff format --check .
uv run mypy --config-file mypy.ini
uv run pytest -m "not integration_live"

Build package artifacts:

uv build

Run the package smoke script from an installed environment:

python3 tools/package_smoke.py --project-root . --dpone-cmd dpone

CI and releases

The OSS repository uses GitHub Actions as the primary automation path. See CI/CD for the workflow map, detailed runbooks, artifacts, and developer guidance.

Key workflows:

.github/workflows/ci.yml runs linting, formatting checks, type checks, tests, coverage, package build, and PostgreSQL XMin integration.
.github/workflows/pages.yml builds and deploys the GitHub Pages documentation site from master.
.github/workflows/release.yml builds and publishes tagged releases to PyPI.
.github/workflows/integration-matrix.yml and .github/workflows/connector-certification.yml provide manual/scheduled production-confidence gates.

Release tags use the format vX.Y.Z, for example:

git tag -a vX.Y.Z -m "Release vX.Y.Z"
git push origin vX.Y.Z

Tagged releases use PyPI Trusted Publishing exclusively. The ordinary release workflow never reads a PyPI API token. The OIDC-only release controller is the sole publisher: it rebuilds only the immutable release tag and verifies the public archive bytes after publication.

Security

Never commit API tokens, PyPI tokens, GitHub tokens, Vault credentials, service-account JSON, or live vendor credentials. If a secret is ever pasted into an issue, chat, commit, or CI log, revoke it before publishing or pushing public history.

See Security policy for the vulnerability reporting process.

License

dpone is licensed under the Apache License 2.0. See LICENSE.

Release files for dpone 0.79.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dpone 0.79.1
File	Size	Uploaded
dpone-0.79.1.tar.gz	4.4 MB	Sep 13, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for dpone 0.79.1
File	Interpreter	ABI	Platform	Reset
dpone-0.79.1-py3-none-any.whl 6.6 MB Sep 13, 2026	Python 3	none	any	Details

Total release size: 10.9 MB

Release files / dpone-0.79.1.tar.gz

Download URL	dpone-0.79.1.tar.gz
Size	4.4 MB
Tags	Source
SHA-256 checksum How to use checksums	`a6ce33b484701df1c0c569384aba59b9e92bbde268ee903741e1d59b49d7a275`
BLAKE2b-256 checksum How to use checksums	`82933a86ec81fa3e28dd5ec14eb1f678e96d1fcd2a75ada6699622ccdc1453a7`
Upload date	Sep 13, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/6.1.0 CPython/3.13.13`

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/PaulKov/dpone-release-controller
Publishing commit: github.com/PaulKov/dpone-release-controller/tree/3b6ff638e2dbefc092447c584084a75ac72fd117

Workflow

Publishing configuration: github.com/PaulKov/dpone-release-controller/blob/3b6ff638e2dbefc092447c584084a75ac72fd117/.github/workflows/pypi-release.yml
Publishing logs: github.com/PaulKov/dpone-release-controller/actions/runs/34760574562/attempts/1

Artifact

Subject: dpone-0.79.1.tar.gz
SHA-256 checksum: a6ce33b484701df1c0c569384aba59b9e92bbde268ee903741e1d59b49d7a275 Verifying attestations

Release files / dpone-0.79.1-py3-none-any.whl

Download URL	dpone-0.79.1-py3-none-any.whl
Size	6.6 MB
Tags	Python 3
SHA-256 checksum How to use checksums	`6654f1e1b9228768cd93eac1e370b00f7e04c0878b7cd9d3fe9fa0b047d588b1`
BLAKE2b-256 checksum How to use checksums	`722a67e2bb0a359c2f290541a2274c37fd194d499c14ed0f3de690e69b6f1171`
Upload date	Sep 13, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/6.1.0 CPython/3.13.13`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/PaulKov/dpone-release-controller
Publishing commit: github.com/PaulKov/dpone-release-controller/tree/3b6ff638e2dbefc092447c584084a75ac72fd117

Workflow

Publishing configuration: github.com/PaulKov/dpone-release-controller/blob/3b6ff638e2dbefc092447c584084a75ac72fd117/.github/workflows/pypi-release.yml
Publishing logs: github.com/PaulKov/dpone-release-controller/actions/runs/34760574562/attempts/1

Artifact

Subject: dpone-0.79.1-py3-none-any.whl
SHA-256 checksum: 6654f1e1b9228768cd93eac1e370b00f7e04c0878b7cd9d3fe9fa0b047d588b1 Verifying attestations