dtex
dtex ("data extraction tool") is an open-source, self-hosted Python extract-load (EL) tool. It moves data from a source (an API, a database, a file drop) into a destination (a warehouse, a database, an object store) — and nothing more. Transformation is dbt's job.
The pitch in one line: a CLI-first, dbt-shaped extract-load tool — pipelines are configs, connectors are folders, no UI blackbox. The #1 principle is to keep it as simple as possible.
Install
pip install dtex # every baked connector, ready
pip install 'dtex[gcs,s3]' # add gs:// / s3:// filesystem reads
pip install 'dtex[gcp-secrets]' # add the GCP Secret Manager resolver
pip install 'dtex[aws-secrets]' # add the AWS Secrets Manager resolver
pip install 'dtex[vault]' # add the HashiCorp Vault resolver
pip install dtex ships every baked source and destination — DuckDB,
BigQuery, the filesystem source's local + Parquet path, the REST / Postgres
/ ShipHero / Stripe / Google Sheets / Google Drive sources, the engine, the CLI. Extras stay opt-in for the
cloud-storage paths of the filesystem source (gs:// / s3://) and for
secret managers (only relevant if your profiles.yml uses secret:// URLs).
dtex requires Python 3.11+. It installs both a CLI (dtex) and an importable
library (import dtex).
Usage
dtex init my_project # scaffold a project
cd my_project
dtex new source my_api # scaffold a source connector
dtex new config my_pipeline # scaffold a pipeline config
dtex validate # check everything
dtex run -p my_pipeline # run the pipeline
dtex runs list -p my_pipeline # show recent run history
A pipeline is one config file binding a source + a destination + a target +
params. Run it with dtex run -p <config>. The library equivalent is
dtex.run(config="my_pipeline") and returns a structured RunResult.
Pre-baked connectors
Each connector ships with its own README — click through for the auth checklist, scope requirements, schema, and known limitations.
Sources:
| Connector | What it does |
|---|---|
filesystem |
CSV / JSONL / Parquet from local, GCS, or S3 |
google_sheets |
Google Sheets — every tab its own table (discovered at run time), or chosen tabs and A1 ranges; typed dates; ADC or service account |
google_drive |
CSV / XLSX / Google Sheets files in a Drive folder (incl. shared drives), incremental by file |
rest |
Paginated REST APIs — 4 pagination strategies, 4 auth modes |
postgres |
Keyset pagination, no OFFSET |
shiphero |
GraphQL |
stripe |
REST resource-as-stream + opt-in Sigma SQL-as-stream |
revenuecat |
v2 API — period-grain subscription transactions, subscriptions, customers, products, daily chart metrics, with a built-in completeness check |
chargebacks911 |
CBAPIv2 — prevention alerts + chargeback cases, incremental |
gads |
Google Ads — GAQL query-as-stream + MCC account auto-discovery |
singular |
Singular.net Reporting API — daily network + agency-tagged attribution reports |
intercom |
Intercom REST API — contacts, conversations + transcript parts, tickets, companies, articles, workspace dimensions; Airbyte-compatible column shapes |
meta |
Meta (Facebook / Instagram) Ads — ad-level daily + campaign-level hourly insights via async report jobs. Created by Auste Luksaite |
konnektive |
Konnektive (Checkout Champ) CRM — orders, transactions, purchases, customers + daily summary; incremental on dateUpdated, resumable backfill |
Destinations:
| Connector | What it does |
|---|---|
duckdb |
Zero-config dev default, all 5 capabilities |
bigquery |
Production warehouse — Parquet-staged via GCS + LOAD jobs, MERGE upserts, cursor-based partitioning |
klaviyo |
Bulk event submission with stable identifiers, bounded retries, and separate DuckDB or BigQuery delivery state |
Engine: per-stream commit + atomic transactions (rollback on failure),
state in the destination's _dtex_state table, run records in _dtex_runs,
structured JSON-lines logs per run, secret redaction, schema evolution
(evolve default, strict opt-in), pipeline-level parallelism with
per-destination caps.
Secret managers: GCP Secret Manager, AWS Secrets Manager, HashiCorp Vault — each as an opt-in extra.
Bundled Claude skills
dtex ships three Claude skills inside the wheel:
dtex-write-config, dtex-write-connector, dtex-debug. They teach
Claude (or any other agent runtime that reads .claude/skills/) the
post-streams-redesign config schema, the @stream decorator pattern, and
the debugging playbook. Install them into a dtex project with:
dtex skills install # copies into ./.claude/skills/dtex/
The first dtex command run inside a project that lacks installed skills
prints a one-line hint (suppressed thereafter). Run dtex skills list to
see which ship with this dtex version and which are installed locally.
If you're not using Claude, just ignore the hint — the skills are inert markdown until installed.
Documentation
The full design handbook lives in docs/.
Start with
00 — Vision & Naming,
02 — Architecture,
06 — Project Anatomy,
12 — Configs, and
10 — Roadmap and Scope.
Security · Contributing · Code of Conduct
- Security policy — how to report a vulnerability.
- Contributing — dev setup, PR process, how to add a connector.
- Code of Conduct.
- Changelog.
License
Apache License 2.0 — see LICENSE.
1 maintainer
Metadata
1 maintainer
Release files for dtex 0.21.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| dtex-0.21.0.tar.gz | 887.8 kB | With provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| dtex-0.21.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 1.5 MB
Release files / dtex-0.21.0.tar.gz
| Download URL | dtex-0.21.0.tar.gz |
|---|---|
| Size | 887.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a26621dacb0fc71fb8216bdf9e1eb131cefbd7e907145ec422444816abc0370b
|
|
BLAKE2b-256 checksum How to use checksums |
fd41e9a4f918ab90433f01d7dd892e195ceb10cbb7d2ae3db169f29f00360321
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 14, 2026.
Transparency logRelease files / dtex-0.21.0-py3-none-any.whl
| Download URL | dtex-0.21.0-py3-none-any.whl |
|---|---|
| Size | 635.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b7063862e4fe05a04c23950b751ad5d0889a3e44fafe66e00236286f49fbd6d7
|
|
BLAKE2b-256 checksum How to use checksums |
8f48746856daf16f8eeac0ff4721df298361af83c004a63309018ee87e768168
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 14, 2026.
Transparency logSecurity for dtex 0.21.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 2 files in this release have been uploaded with provenance metadata.
Attestation for dtex-0.21.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 14, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
