dpone

PyPI Python CI Docs License

dpone is a Python ETL framework for declarative, YAML-driven data pipelines. It helps data teams describe sources, sinks, load strategies, dependencies, conventions, and operational checks as reusable configuration instead of one-off scripts.

The public package name, import name, GitHub repository name, and CLI name are all intentionally short: dpone.

Repository: https://github.com/PaulKov/dpone

Public install:

python3 -m pip install dpone
dpone --help

First Airflow DAG

Create and verify a provider-loadable DAG without writing Airflow Python or configuring credentials:

dpone init project --airflow
dpone init pipeline orders_daily --recipe mssql-to-clickhouse-incremental
dpone check orders_daily
dpone airflow preview orders_daily
dpone test orders_daily

The five commands are the complete offline beginner journey. A live, bounded sample is a separate platform-gated action. See First Airflow DAG.

Documentation map

Start here if you are evaluating or operating dpone:

What dpone gives you

  • YAML manifests for single-process and batch ETL definitions.
  • Built-in DAG/dependency inspection for pipeline debugging.
  • Runtime abstractions for sources, sinks, connectors, state, reconciliation, and safe SQL logging.
  • Optional integrations for PostgreSQL, MSSQL/SQL Server, ClickHouse, BigQuery/GCS, Kafka, pandas, Google Ads, and HashiCorp Vault.
  • A CLI designed for self-service validation, rendering, explainability, and documentation checks.
  • Compatibility shims for older import paths while the canonical package layout continues to stabilize.

Installation

Install the core package from PyPI:

pip install dpone
dpone --version
dpone -v

Install common extras for local ETL development:

pip install "dpone[postgres,mssql,clickhouse,kafka,gcp,s3,azure,pandas,vault]"

Install everything currently published by the project:

pip install "dpone[full]"

With uv:

uv add dpone
uv add "dpone[full]"

Install the Kubernetes SDK only in the restricted platform environment that runs Airflow Connection Secret GC or runtime-Pod retention:

pip install "dpone[kubernetes]"

Optional extras

Extra Purpose
postgres PostgreSQL connectivity via psycopg
mssql Microsoft SQL Server connectivity via pyodbc; production bulk paths use external ODBC Driver 18 and bcp
clickhouse ClickHouse connectivity
gcp Google BigQuery and Google Cloud Storage support
s3 AWS S3 object storage staging support via boto3
azure Azure Blob Storage plus workload identity via azure-storage-blob and azure-identity
object_storage S3, GCS, and Azure Blob staging, including Azure workload identity
kafka Kafka batch source/sink support via confluent-kafka, Schema Registry codecs, Avro, JSON Schema, and Protobuf helpers
pandas DataFrame-based extract/load helpers
vault HashiCorp Vault integration via public vault-kv-client
kubernetes Metadata-only, namespace-scoped Airflow Connection Secret GC and runtime-Pod retention for platform operators
google_ads Google Ads API support
full All public extras above

Vault support uses vault-kv-client, published on PyPI as vault-kv-client. dpone imports the canonical vault_kv_client module only; legacy Vault import paths are not supported by dpone.

Credentials can come from environment variables, Airflow Connections, HashiCorp Vault-compatible KV, or inline params for smoke tests. See Connections and credentials for copy-paste examples for Postgres, MSSQL, ClickHouse, BigQuery, Kafka, and REST API.

Quick start

Create a batch manifest, for example examples/batch/landing_postgres_to_bq.batch.yaml:

# yaml-language-server: $schema=../../src/dpone/schema/etl-batch-manifest.schema.json
kind: dpone.batch.v1
convention: landing_raw_v1
registry: ../registry/sources.yaml

vars:
  src_system: demo_source
  src_database: demo_db
  owner_team: data-platform
  owner_contact: data-platform@example.com
  sla: daily

defaults:
  source:
    type: postgres
    connection_type: vault
    connection_id: postgres-demo
    vault_path: postgres/demo-source
    options:
      batch_size: 100000
      export_format: csv

  sink:
    type: bigquery
    connection_type: vault
    connection_id: bigquery-demo
    vault_path: gcp/demo-project-prod/bq/service-account
    staging:
      schema: stg
    strategy:
      mode: full_refresh
      overwrite_type: exchange

schemas:
  public:
    tables:
      - core_city

Validate and render it:

dpone manifest validate examples/batch/landing_postgres_to_bq.batch.yaml \
  --profile landing_raw_v1 \
  --registry examples/registry/sources.yaml

dpone manifest render examples/batch/landing_postgres_to_bq.batch.yaml \
  --selector public.core_city \
  --registry examples/registry/sources.yaml

Inspect pipeline dependencies:

dpone dag report examples/batch/landing_postgres_to_bq.batch.yaml \
  --base-path . \
  --format json \
  --preset ci \
  --registry examples/registry/sources.yaml

CLI overview

dpone --help
dpone manifest --help
dpone dag --help
dpone docs --help

Common commands:

dpone manifest list examples/batch/landing_postgres_to_bq.batch.yaml
dpone manifest validate examples/batch/landing_postgres_to_bq.batch.yaml --recursive
dpone manifest render examples/batch/landing_postgres_to_bq.batch.yaml --selector public.core_city
dpone manifest explain examples/batch/landing_postgres_to_bq.batch.yaml --selector public.core_city --why sink.table.schema
dpone dag list-edges examples/batch/landing_postgres_to_bq.batch.yaml --with-groups --with-refs
dpone dag explain-node examples/batch/landing_postgres_to_bq.batch.yaml --task public.core_city
dpone dag report examples/batch/landing_postgres_to_bq.batch.yaml --preset ci --format md

Repository layout

src/dpone/      Python package source code
docs/           User and developer documentation
examples/       Public example manifests and registries
tests/          Unit and integration tests
tools/          Local smoke and release helper scripts

Canonical imports live under:

  • dpone.manifest.*
  • dpone.dag.*
  • dpone.runtime.*
  • dpone.contracts.*
  • dpone.ports.*
  • dpone.adapters.*

Legacy paths such as dpone.core.*, dpone.lib.*, dpone.source.*, and dpone.sink.* are compatibility shims. Prefer canonical imports for new code.

Local development

uv sync --all-extras
uv run ruff check .
uv run ruff format --check .
uv run mypy --config-file mypy.ini
uv run pytest -m "not integration_live"

Build package artifacts:

uv build

Run the package smoke script from an installed environment:

python3 tools/package_smoke.py --project-root . --dpone-cmd dpone

CI and releases

The OSS repository uses GitHub Actions as the primary automation path. See CI/CD for the workflow map, detailed runbooks, artifacts, and developer guidance.

Key workflows:

  • .github/workflows/ci.yml runs linting, formatting checks, type checks, tests, coverage, package build, and PostgreSQL XMin integration.
  • .github/workflows/pages.yml builds and deploys the GitHub Pages documentation site from master.
  • .github/workflows/release.yml builds and publishes tagged releases to PyPI.
  • .github/workflows/integration-matrix.yml and .github/workflows/connector-certification.yml provide manual/scheduled production-confidence gates.

Release tags use the format vX.Y.Z, for example:

git tag -a vX.Y.Z -m "Release vX.Y.Z"
git push origin vX.Y.Z

Tagged releases use PyPI Trusted Publishing exclusively. The ordinary release workflow never reads a PyPI API token. The OIDC-only release controller is the sole publisher: it rebuilds only the immutable release tag and verifies the public archive bytes after publication.

Security

Never commit API tokens, PyPI tokens, GitHub tokens, Vault credentials, service-account JSON, or live vendor credentials. If a secret is ever pasted into an issue, chat, commit, or CI log, revoke it before publishing or pushing public history.

See Security policy for the vulnerability reporting process.

License

dpone is licensed under the Apache License 2.0. See LICENSE.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for paulkov007 from gravatar.com paulkov007

Credits

Author/Maintainer: PaulKov

License expression

Apache-2.0
View SPDX License List

Requires

Python <3.13, >=3.11

Provides Extra

accel azure clickhouse columnar dbt dbt-mssql full gcp google-ads kafka kubernetes mssql mysql object-storage pandas postgres s3 semantic-refresh-airflow vault

Tags

etl data-platform data-engineering yaml dag bigquery postgres mssql sql-server clickhouse kafka

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dpone-0.74.33.tar.gz (4.1 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

dpone-0.74.33-py3-none-any.whl (6.2 MB view details)

Uploaded Python 3

File details

Details for the file dpone-0.74.33.tar.gz.

File metadata

  • Download URL: dpone-0.74.33.tar.gz
  • Upload date:
  • Size: 4.1 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.13

File hashes

Hashes for dpone-0.74.33.tar.gz
Algorithm Hash digest
SHA256 1fed111eedcacefb2c5a30d61abf5d587bb7705b4bb00025c68e380f471362f7
MD5 184a53f4ee774a4dcf4fa02f44b11896
BLAKE2b-256 90f74ad993aedd50094f24a5d369406ef8aab05d1d4f3962f5bc6c586b19d9cd

See more details on using hashes here.

Provenance

The following attestation bundles were made for dpone-0.74.33.tar.gz:

Publisher: pypi-release.yml on PaulKov/dpone-release-controller

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dpone-0.74.33-py3-none-any.whl.

File metadata

  • Download URL: dpone-0.74.33-py3-none-any.whl
  • Upload date:
  • Size: 6.2 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.13

File hashes

Hashes for dpone-0.74.33-py3-none-any.whl
Algorithm Hash digest
SHA256 127c25325f62a5b26ae141045f39c625ea99de34fd025a79cd443693ee7b7110
MD5 ebc5de088fa73ccd48065f2bf444400d
BLAKE2b-256 43794df7535191602a667eea8472db2a76f496a83704a602629b7c210c4329a9

See more details on using hashes here.

Provenance

The following attestation bundles were made for dpone-0.74.33-py3-none-any.whl:

Publisher: pypi-release.yml on PaulKov/dpone-release-controller

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.74.33 This release

2 files

0.74.32

2 files

0.74.31

2 files

0.74.30

2 files

0.74.29

2 files

0.74.28

2 files

0.74.27

2 files

0.74.26

2 files

0.74.25

2 files

0.74.24

2 files

0.74.23

2 files

0.74.22

2 files

0.74.20

2 files

0.74.19

2 files

0.74.17

2 files

0.74.16

2 files

0.74.15

2 files

0.74.14

2 files

0.74.13

2 files

0.74.12

2 files

0.74.11

2 files

0.74.10

2 files

0.74.9

2 files

0.74.8

2 files

0.74.7

2 files

0.74.6

2 files

0.74.5

2 files

0.74.4

2 files

0.74.3

2 files

0.74.2

2 files

0.74.0

2 files

0.73.42

2 files

0.73.38

2 files

0.73.36

2 files

0.73.34

2 files

0.73.31

2 files

0.73.30

2 files

0.73.29

2 files

0.73.28

2 files

0.73.25

2 files

0.73.24

2 files

0.73.23

2 files

0.73.22

2 files

0.73.21

2 files

0.73.20

2 files

0.73.19

2 files

0.73.17

2 files

0.73.16

2 files

0.73.15

2 files

0.73.14

2 files

0.73.13

2 files

0.73.10

2 files

0.73.9

2 files

0.73.6

2 files

0.73.2

2 files

0.73.1

2 files

0.73.0

2 files

0.72.5

2 files

0.72.4

2 files

0.72.3

2 files

0.72.2

2 files

0.72.1

2 files

0.72.0

2 files

0.70.0

2 files

0.69.0

2 files

0.68.25

2 files

0.68.24

2 files

0.68.23

2 files

0.68.20

2 files

0.68.19

2 files

0.68.18

2 files

0.68.17

2 files

0.68.16

2 files

0.68.15

2 files

0.68.14

2 files

0.68.12

2 files

0.68.11

2 files

0.68.10

2 files

0.68.8

2 files

0.68.7

2 files

0.68.6

2 files

0.68.5

2 files

0.68.4

2 files

0.68.2

2 files

0.68.1

2 files

0.68.0

2 files

0.67.0

2 files

0.66.17

2 files

0.66.16

2 files

0.66.15

2 files

0.66.14

2 files

0.66.13

2 files

0.66.12

2 files

0.66.11

2 files

0.66.10

2 files

0.66.9

2 files

0.66.8

2 files

0.66.3

2 files

0.66.2

2 files

0.66.1

2 files

0.66.0

2 files

0.65.2

2 files

0.65.1

2 files

0.65.0

2 files

0.64.1

2 files

0.64.0

2 files

0.63.0

2 files

0.62.1

2 files

0.62.0

2 files

0.61.3

2 files

0.61.2

2 files

0.61.1

2 files

0.61.0

2 files

0.60.2

2 files

0.60.1

2 files

0.60.0

2 files

0.59.0

2 files

0.58.0

2 files

0.57.0

2 files

0.56.0

2 files

0.55.0

2 files

0.54.0

2 files

0.53.0

2 files

0.52.0

2 files

0.51.0

2 files

0.50.0

2 files

0.49.0

2 files

0.48.1

2 files

0.48.0

2 files

0.47.0

2 files

0.46.0

2 files

0.45.0

2 files

0.44.0

2 files

0.43.0

2 files

0.42.0

2 files

0.41.0

2 files

0.40.1

2 files

0.40.0

2 files

0.39.0

2 files

0.38.0

2 files

0.37.1

2 files

0.37.0

2 files

0.36.0

2 files

0.34.0

2 files

0.33.2

2 files

0.33.1

2 files

0.33.0

2 files

0.32.3

2 files

0.32.1

2 files

0.32.0

2 files

0.31.1

2 files

0.31.0

2 files

0.29.0

2 files

0.28.0

2 files

0.27.0

2 files

0.26.0

2 files

0.25.0

2 files

0.24.2

2 files

0.24.1

2 files

0.24.0

2 files

0.23.1

2 files

0.23.0

2 files

0.22.0

2 files

0.21.0

2 files

0.20.0

2 files

0.19.0

2 files

0.18.0

2 files

Pre-release

0.18.0rc1

2 files

0.17.0

2 files

0.16.3

2 files

0.16.2

2 files

0.16.1

2 files

0.16.0

2 files

0.15.2

2 files

0.15.1

2 files

0.15.0

2 files

0.14.4

2 files

0.14.3

2 files

0.14.2

2 files

0.14.1

2 files

0.14.0

2 files

0.13.0

2 files

0.12.0

2 files

0.11.0

2 files

0.10.0

2 files

0.9.0

2 files

0.8.0

2 files

0.7.10

2 files

0.7.9

2 files

0.7.8

2 files

0.7.7

2 files

0.7.6

2 files

0.7.5

2 files

0.7.4

2 files

0.7.3

2 files

0.7.2

2 files

0.7.1

2 files

0.7.0

2 files

0.6.0

2 files

0.5.1

2 files

0.5.0

2 files

0.4.1

2 files

0.4.0

2 files

0.3.0

2 files

0.2.8

2 files

0.2.7

2 files

0.2.6

2 files

0.2.5

2 files

0.2.4

2 files

0.2.3

2 files

0.2.2

2 files

0.2.1

2 files

0.2.0

2 files

0.1.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page