dcf-core 1.1.1

D.ata C.ollection F.ramework - Define a data source, run dcf, query the data

dcf

D.ata C.ollection F.ramework

uvx --from dcf-core dcf init

---

How it works

1. Define a data collector in YAML — source, schema, cadence
2. Run it with dcf run
3. Query data from your data lake

---

▶️ Video Tutorial: Run a DCF Collector with GitHub API

---

Quickstart

Get real data. From an API. Into your Lakehouse. Query it with SQL. In 5 lines.

mkdir dcf-demo && cd dcf-demo
uvx --from dcf-core dcf init
uv sync
uv run dcf run so_questions
uv run dcf query 'SELECT * FROM stackoverflow.so_questions'

dcf init creates pyproject.toml, profiles.yml, .gitignore, collectors/, and an example collector.

---

Example

dcf collector

name: so_questions
namespace: stackoverflow

source:
  type: http
  url: https://api.stackexchange.com/2.3/questions
  response:
    records_path: items
  params:
    - name: site
      type: string
      value: stackoverflow
    - name: fromdate
      type: string
      format: "%s"
    - name: todate
      type: string
      format: "%s"
  schema:
    columns:
      - name: question_id
        path: question_id
        type: integer
      - name: title
        path: title
        type: string
      - name: creation_date
        path: creation_date
        type: integer

cadence:
  strategy: incremental
  primary_key: question_id
  iterate:
    - type: date_range
      params: [fromdate, todate]
      start: "2025-01-01"
      end: today
      step: 30 days

dcf run

uv run dcf run so_questions

dcf query

uv run dcf query 'SELECT * FROM stackoverflow.so_questions LIMIT 5'

---

More features

• Importing collectors from dcf-hub, GitHub, or PyPI — dcf import
• Authenticated collectors — bearer tokens, API keys, {{ env.VAR }}
• Collector config reference — full YAML field reference

---

Contributing

git clone https://github.com/zephschafer/dcf && cd dcf && uv sync

Point a local project at your checkout:

[tool.uv.sources]
dcf-core = { path = "../dcf", editable = true }

To verify changes:

uv run dcf run so_questions
uv run dcf query 'SELECT * FROM stackoverflow.so_questions'

Releasing: bump version in pyproject.toml and push to main — GitHub Actions publishes to PyPI automatically.

---

Package structure

dcf/
├── cli.py              Entry point (Typer)
├── config/
│   ├── models.py       Pydantic models for collector YAML
│   └── loader.py       YAML loading + env var resolution
├── engine/
│   ├── runner.py       Outer loop (iterate → fetch → project → write)
│   ├── fetcher.py      HTTP and Python source fetchers
│   ├── iterator.py     Date range and categorical iteration
│   ├── projector.py    Schema projection and path extraction
│   └── transforms.py   Column transforms
├── writer/
│   └── iceberg.py      Write strategies (incremental / append / full_refresh)
└── gcp/                GCP auth, provisioning, Terraform wrappers