taxi-seed

taxi-seed is one repo, one Python package with five tools (plus a shared library) for working with the NYC Taxi & Limousine Commission (TLC) trip record data set: a WAF-aware CloudFront downloader (WAF = Web Application Firewall — the layer AWS uses to block traffic that looks like a scraper), a schema-drift analyzer, a normalize step that halts on any data loss unless the operator explicitly acknowledges the drift, a loader that lands normalized parquet into a target database, and an orchestrator that drives the whole download → normalize → load pipeline end-to-end. The project is MIT-licensed. The downloader was loosely based on toddwschneider/nyc-taxi-data; the other tools are original to this repo. See Acknowledgments for details.

• WAF-aware CloudFront downloader with a 30s / 90s / 270s exponential backoff ladder (capped at 3600s) and stop-on-local incremental catch-up so nightly crons stay cheap once the mirror is warm.
• Normalizer that treats data loss as a first-class error — missing columns, lossy casts, and silent renames halt the run; explicit ack_date acknowledgment is required before drift is written through.
• Loader + orchestrator that land normalized parquet into a target database and drive the full pipeline end-to-end on a schedule.

→ Full documentation: https://andrekamman.github.io/taxi-seed/

The site has the deep-dive guides, a cookbook of cross-cutting recipes, an architecture overview, and the design specs; this README stays intentionally short.

Components

One repo, five tools plus a shared library:

• downloader/ — Python CLI (taxi-download) that mirrors the TLC CloudFront bucket to local parquet.
• schema-drift/ — Python CLI that reports column-name and column-shape drift across a mirror.
• normalize/ — Python CLI that rewrites a mirror to a single target schema, refusing to lose data.
• loader/ — Python CLI that loads normalized parquet into a target database.
• orchestrator/ — Python CLI that drives download → normalize → load as one pipeline.
• shared/ — common library code (sql_generator.py for CREATE TABLE DDL generation, type_mapping.py for DuckDB→SQL Server type mapping) used across the tools.

Each component has a short README.md that points at the guide on the site; the guide is authoritative.

Quick start

git clone https://github.com/andrekamman/taxi-seed.git
cd taxi-seed
uv sync
uv run taxi-download yellow --recent 3

Downloads ~200 MB in 1–2 minutes on residential broadband. This Quick Start only exercises the downloader. All tools live in one uv sync-managed environment; the Getting Started tutorial walks the full end-to-end path from clone to normalized parquet.

A full-history mirror is roughly 40–100 GB (depending on how many of the four series and how much history you mirror) and takes 6–10 hours end-to-end; the downloader is designed to be resumable, incremental, and cheap to re-run on a schedule rather than something you kick off once and hope survives.

Requirements

• Python 3.12 or 3.13, uv. Required by every tool in the repo, including the downloader.
• Disk sized to intent — see the Downloader guide.
• Individual tools list per-guide prerequisites (a target database for the loader, etc.).

Everything runs on macOS, Linux, and Windows (via Git Bash). CI runs the test suite and the strict docs build on every PR.

Documentation map

The documentation site is the source of truth. High-level sections:

• Getting Started — 10-minute end-to-end walkthrough from clone to normalized parquet.
• Guides — one deep-dive per tool (Downloader, Schema Drift, Normalize).
• Cookbook — cross-cutting recipes (nightly cron, DuckDB httpfs querying, corporate proxy).
• Architecture — how the tools fit together end-to-end.
• Reference — configuration keys and exit codes.
• Design Specs — the "why is this shaped this way?" trail.

Contributing

PRs welcome. See docs/contributing.md — also published as the Contributing page on the site — for the dev-setup, test, and PR-checklist details.

License

MIT — see LICENSE.

Acknowledgments

The downloader was loosely based on toddwschneider/nyc-taxi-data (MIT) — specifically its convention for organizing TLC parquet by type and year. The rest of the repo (schema-drift analyzer, normalizer, loader, orchestrator) is original work and shares no code with Todd's project. See THIRD_PARTY_NOTICES for the full attribution.