dpone
dpone is a Python ETL framework for declarative, YAML-driven data pipelines. It helps data teams describe sources, sinks, load strategies, dependencies, conventions, and operational checks as reusable configuration instead of one-off scripts.
The public package name, import name, GitHub repository name, and CLI name are all intentionally short: dpone.
Repository: https://github.com/PaulKov/dpone
Public install:
python3 -m pip install dpone
dpone --help
First Airflow DAG
Create and verify a provider-loadable DAG without writing Airflow Python or configuring credentials:
dpone init project --airflow
dpone init pipeline orders_daily --recipe mssql-to-clickhouse-incremental
dpone check orders_daily
dpone airflow preview orders_daily
dpone test orders_daily
The five commands are the complete offline beginner journey. A live, bounded sample is a separate platform-gated action. See First Airflow DAG.
Documentation map
Start here if you are evaluating or operating dpone:
- Documentation index
- CLI reference
- Running pipelines from CLI and Python
- Connector overview
- Source -> sink matrix
- Manual integration matrix
- CI/CD
- Testing runbooks
- Runtime observability
- Runtime decision audit
- Object storage staging
- Airflow artifact trust (fail-closed preview)
- Airflow exact-cache operations and recovery
- Airflow Kubernetes cache deployment
- Airflow runtime Pod retention
- dbt self-service platform workflows
- Supply-chain evidence
- Release evidence
- Load strategies
- Nested normalization
- Load lineage
- Type mapping matrix
- Schema evolution
- Production readiness
- Architecture
What dpone gives you
- YAML manifests for single-process and batch ETL definitions.
- Built-in DAG/dependency inspection for pipeline debugging.
- Runtime abstractions for sources, sinks, connectors, state, reconciliation, and safe SQL logging.
- Optional integrations for PostgreSQL, MSSQL/SQL Server, ClickHouse, BigQuery/GCS, Kafka, pandas, Google Ads, and HashiCorp Vault.
- A CLI designed for self-service validation, rendering, explainability, and documentation checks.
- Compatibility shims for older import paths while the canonical package layout continues to stabilize.
Installation
Install the core package from PyPI:
pip install dpone
dpone --version
dpone -v
Install common extras for local ETL development:
pip install "dpone[postgres,mssql,clickhouse,kafka,gcp,s3,azure,pandas,vault]"
Install everything currently published by the project:
pip install "dpone[full]"
With uv:
uv add dpone
uv add "dpone[full]"
Install the Kubernetes SDK only in the restricted platform environment that runs Airflow Connection Secret GC or runtime-Pod retention:
pip install "dpone[kubernetes]"
Optional extras
| Extra | Purpose |
|---|---|
postgres |
PostgreSQL connectivity via psycopg |
mssql |
Microsoft SQL Server connectivity via pyodbc; production bulk paths use external ODBC Driver 18 and bcp |
clickhouse |
ClickHouse connectivity |
gcp |
Google BigQuery and Google Cloud Storage support |
s3 |
AWS S3 object storage staging support via boto3 |
azure |
Azure Blob Storage plus workload identity via azure-storage-blob and azure-identity |
object_storage |
S3, GCS, and Azure Blob staging, including Azure workload identity |
kafka |
Kafka batch source/sink support via confluent-kafka, Schema Registry codecs, Avro, JSON Schema, and Protobuf helpers |
pandas |
DataFrame-based extract/load helpers |
vault |
HashiCorp Vault integration via public vault-kv-client |
kubernetes |
Metadata-only, namespace-scoped Airflow Connection Secret GC and runtime-Pod retention for platform operators |
google_ads |
Google Ads API support |
full |
All public extras above |
Vault support uses vault-kv-client, published on PyPI as vault-kv-client. dpone imports the canonical vault_kv_client module only; legacy Vault import paths are not supported by dpone.
Credentials can come from environment variables, Airflow Connections, HashiCorp Vault-compatible KV, or inline params for smoke tests. See Connections and credentials for copy-paste examples for Postgres, MSSQL, ClickHouse, BigQuery, Kafka, and REST API.
Quick start
Create a batch manifest, for example examples/batch/landing_postgres_to_bq.batch.yaml:
# yaml-language-server: $schema=../../src/dpone/schema/etl-batch-manifest.schema.json
kind: dpone.batch.v1
convention: landing_raw_v1
registry: ../registry/sources.yaml
vars:
src_system: demo_source
src_database: demo_db
owner_team: data-platform
owner_contact: data-platform@example.com
sla: daily
defaults:
source:
type: postgres
connection_type: vault
connection_id: postgres-demo
vault_path: postgres/demo-source
options:
batch_size: 100000
export_format: csv
sink:
type: bigquery
connection_type: vault
connection_id: bigquery-demo
vault_path: gcp/demo-project-prod/bq/service-account
staging:
schema: stg
strategy:
mode: full_refresh
overwrite_type: exchange
schemas:
public:
tables:
- core_city
Validate and render it:
dpone manifest validate examples/batch/landing_postgres_to_bq.batch.yaml \
--profile landing_raw_v1 \
--registry examples/registry/sources.yaml
dpone manifest render examples/batch/landing_postgres_to_bq.batch.yaml \
--selector public.core_city \
--registry examples/registry/sources.yaml
Inspect pipeline dependencies:
dpone dag report examples/batch/landing_postgres_to_bq.batch.yaml \
--base-path . \
--format json \
--preset ci \
--registry examples/registry/sources.yaml
CLI overview
dpone --help
dpone manifest --help
dpone dag --help
dpone docs --help
Common commands:
dpone manifest list examples/batch/landing_postgres_to_bq.batch.yaml
dpone manifest validate examples/batch/landing_postgres_to_bq.batch.yaml --recursive
dpone manifest render examples/batch/landing_postgres_to_bq.batch.yaml --selector public.core_city
dpone manifest explain examples/batch/landing_postgres_to_bq.batch.yaml --selector public.core_city --why sink.table.schema
dpone dag list-edges examples/batch/landing_postgres_to_bq.batch.yaml --with-groups --with-refs
dpone dag explain-node examples/batch/landing_postgres_to_bq.batch.yaml --task public.core_city
dpone dag report examples/batch/landing_postgres_to_bq.batch.yaml --preset ci --format md
Repository layout
src/dpone/ Python package source code
docs/ User and developer documentation
examples/ Public example manifests and registries
tests/ Unit and integration tests
tools/ Local smoke and release helper scripts
Canonical imports live under:
dpone.manifest.*dpone.dag.*dpone.runtime.*dpone.contracts.*dpone.ports.*dpone.adapters.*
Legacy paths such as dpone.core.*, dpone.lib.*, dpone.source.*, and dpone.sink.* are compatibility shims. Prefer canonical imports for new code.
Local development
uv sync --all-extras
uv run ruff check .
uv run ruff format --check .
uv run mypy --config-file mypy.ini
uv run pytest -m "not integration_live"
Build package artifacts:
uv build
Run the package smoke script from an installed environment:
python3 tools/package_smoke.py --project-root . --dpone-cmd dpone
CI and releases
The OSS repository uses GitHub Actions as the primary automation path. See CI/CD for the workflow map, detailed runbooks, artifacts, and developer guidance.
Key workflows:
.github/workflows/ci.ymlruns linting, formatting checks, type checks, tests, coverage, package build, and PostgreSQL XMin integration..github/workflows/pages.ymlbuilds and deploys the GitHub Pages documentation site frommaster..github/workflows/release.ymlbuilds and publishes tagged releases to PyPI.- .github/workflows/integration-matrix.yml and .github/workflows/connector-certification.yml provide manual/scheduled production-confidence gates.
Release tags use the format vX.Y.Z, for example:
git tag -a vX.Y.Z -m "Release vX.Y.Z"
git push origin vX.Y.Z
Tagged releases use PyPI Trusted Publishing exclusively. The ordinary release workflow never reads a PyPI API token. Any future emergency token path must be a separate protected break-glass workflow with explicit approval and auditable release evidence.
Security
Never commit API tokens, PyPI tokens, GitHub tokens, Vault credentials, service-account JSON, or live vendor credentials. If a secret is ever pasted into an issue, chat, commit, or CI log, revoke it before publishing or pushing public history.
See Security policy for the vulnerability reporting process.
License
dpone is licensed under the Apache License 2.0. See LICENSE.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file dpone-0.74.8.tar.gz.
File metadata
- Download URL: dpone-0.74.8.tar.gz
- Upload date:
- Size: 3.8 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.2.0 CPython/3.12.11
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
dcd5122b66f67d24d28667553fc8e08243fbaceaab1466aea37f9843686456b5
|
|
| MD5 |
797423840e5fd7b66d85bbdc7425ecce
|
|
| BLAKE2b-256 |
55ccd211dee432e4784dd5cda104480543aa979f7c86e120dbf35e6ff05698a9
|
File details
Details for the file dpone-0.74.8-py3-none-any.whl.
File metadata
- Download URL: dpone-0.74.8-py3-none-any.whl
- Upload date:
- Size: 5.7 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.2.0 CPython/3.12.11
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
6df885bd0b6f52e090a72e13999e271bd3725852c428b7ba9ab43787f6cac8ba
|
|
| MD5 |
2128d15ebee6fb870d08b6fc5cdb245a
|
|
| BLAKE2b-256 |
5ac80ebea3c335a7277bc19b50b158cb6778b2e852cdcbf77eaa7d4ecca83a28
|
