filedge 0.3.0

Atomic file ingestion with content-hash idempotency and a full audit trail — into SQLite, PostgreSQL, BigQuery, Databricks, or DuckDB.

Filedge

Files are the universal building block of data engineering. Whether data starts in Kafka, Stripe's API, a partner SFTP, or a CDC stream, every reliable pipeline eventually crystallizes it into a file before it touches the warehouse. Filedge is the load boundary built around that fact: atomic per-file ingestion, content-hash idempotency, and a full audit trail — into SQLite, PostgreSQL, BigQuery, Databricks, or DuckDB.

Why files?

Streams are continuous; files are discrete. That discreteness is what makes ingestion auditable: a file has a SHA-256, a row count, a state in the audit DB, and a row-level provenance trail in the destination. Every downstream question — did we load this?, replay this, where did this row come from? — has a deterministic anchor.

Filedge starts where the file lands and ends when its rows are committed. Upstream is your choice: dlt or vendor exporters for APIs, Kafka Connect or Vector for queues, rclone for SFTP. Downstream is your warehouse. The hard part in between — retry-safe commits, dedupe, retries, lineage — is all Filedge does.

What it gives you that a hand-rolled DAG doesn't

Failure mode	Typical pipeline	Filedge
Half-written tables after a crash	Manual cleanup	Per-file atomic commit, retry-safe by content hash
"Did we already load this file?"	Filename heuristics	SHA-256 dedupe at the entry point
"Where did this row come from?"	Grep logs	_source_file_hash + _ingested_at on every row
Stale lock from a killed worker	Page someone	Reclaimed automatically on next run
One bad file blocks the pipeline	Skip and forget	Bounded retry → terminal FAILED with audit
Schema drift in destination	Silent corruption	Loud failure with a clear diff

How it differs from neighbors

• vs Airbyte / Fivetran / dlt — those fetch (paginate APIs, manage cursors). Filedge lands — it takes whatever they produce as files and makes the write to the warehouse audit-grade. Use them as Fetchers in front of Filedge.
• vs Kafka Connect / Flink / Spark Structured Streaming — streaming systems own continuous offsets and incremental state. Filedge owns the file as the unit of work — simpler to reason about, replay, and audit. Materialize queues to files, then ingest.
• vs Airflow + custom Python loaders — same DAG shape, but partial-load corruption, lock reclaim, retry caps, idempotent CDC apply, and row provenance are already wired in.
• vs Iceberg / Delta tables — those are table formats. Filedge is what writes to them (or to plain BigQuery / Postgres / Databricks tables) with the per-file commit guarantee.

Quick start

Requires uv.

uv sync --extra dev                          # core (SQLite)
uv sync --extra dev --extra postgres         # + PostgreSQL
uv sync --extra dev --extra bigquery         # + BigQuery
uv sync --extra dev --extra databricks       # + Databricks
uv sync --extra dev --extra duckdb           # + DuckDB
uv sync --extra dev --extra authoring        # + Authoring UI
uv sync --extra dev --extra excel            # + Excel (.xlsx)
uv sync --extra dev --extra kafka            # + Reference Queue Materializer

Declare a pipeline:

# pipeline.yaml
format: csv
dest_table: orders
write_mode: append          # append | truncate | cdc
retry_cap: 3
batch_size: 1000

connector:
  type: sqlite
  url: sqlite:///orders.db

columns:
  - { source: order_id,   dest: order_id,   type: string,  required: true }
  - { source: amount,     dest: amount,     type: float,   required: true }
  - { source: order_date, dest: order_date, type: date }

Run it:

filedge run --dir ./incoming --config pipeline.yaml --audit-db-url sqlite:///filedge.db
# Committed: 3  Failed: 0  Skipped: 0  New: 3  Reclaimed: 0  Retried: 0

filedge status --audit-db-url sqlite:///filedge.db
# PENDING: 0  PROCESSING: 0  COMMITTED: 3  FAILED: 0

Don't know the schema yet? filedge inspect data.csv samples the file and prints a columns: block with confidence tiers ready to paste.

Prefer to author interactively? filedge author data.csv launches a local terminal UI that runs schema inference, lets you review columns, write modes, connectors, and field encryption, validates the result, and writes a ready-to-run pipeline folder. To revise a pipeline later, filedge author --pipeline pipelines/<id> re-opens it in place — or run filedge author with no arguments to browse and pick from the registry. See the author guide.

Pulling from APIs or queues? Use an upstream Fetcher or Queue Materializer to land complete Files, then run Filedge. The first-party companions filedge-fetch and filedge-materialize demonstrate the audited materialize-to-files contract for API Sources, EDGAR companyConcept, and Kafka Queue Sources. See the API sources and queue sources guides.

Connectors

The destination is configured via a connector: block in pipeline.yaml. Built-ins:

Destination	Extra	Notes
SQLite	(core)	Default for local dev; configure with type: sqlite and a url
PostgreSQL	postgres	COPY bulk load; idempotent via per-hash DELETE
BigQuery	bigquery	NDJSON staging + load job; job-ID-keyed idempotency (7-day window)
Databricks	databricks	Unity Catalog volume staging
DuckDB	duckdb	File-based; single-writer, fails fast if locked

See docs/guides/run.md for full connector config, credentials, and live-integration test setup.

How a run works

filedge run
├── Reset FAILED below retry_cap → PENDING
├── Reclaim stale PROCESSING locks → PENDING
├── Connector: ensure destination table exists
├── Hash files in watched dir; enqueue new hashes as PENDING
└── For each PENDING file:
    ├── Audit DB: mark PROCESSING        (distributed lock)
    ├── Connector: stream rows → commit  (idempotent per file_hash)
    └── Audit DB: mark COMMITTED / FAILED

The audit DB and the destination are separate systems. A crash between connector commit and audit mark leaves the file PROCESSING — the next run reclaims it, and the connector's per-hash idempotency guarantees no duplicate rows.

More

• Guides: author · run · scale · inspect · preview · validate · compact · healthcheck · requeue · audit export · CDC files · API sources · queue sources · source manifests
• Release notes: CHANGELOG.md
• Domain model: CONTEXT.md
• Architecture decisions:
  • ADR-0001: Single-transaction commit
  • ADR-0002: Content hash as idempotency key
  • ADR-0003: Strict-mode validation
  • ADR-0004: Audit DB / Connector split
  • ADR-0005: SFTP out of scope
  • ADR-0006: API sources fetched to files
  • ADR-0007: Queue source ingestion model
  • ADR-0008: Schema inference confidence tiers
  • ADR-0009: Warehouse CDC applied-file markers
  • ADR-0010: Audit export static site
  • ADR-0011: Source manifest and lineage
  • ADR-0012: Excel format support
  • ADR-0013: Fixed-width format support
  • ADR-0014: Column-level field encryption
  • ADR-0015: Control and Audit Platform starts with local Pipeline Authoring
  • ADR-0016: Authoring UI — Textual TUI
  • ADR-0017: Pipeline Folder and Pipeline Registry layout
  • ADR-0018: Reference Fetcher external companion
  • ADR-0019: Dead-Letter Quarantine

License

Apache 2.0 — see LICENSE.