taxi-seed 0.1.0

NYC TLC Taxi data analysis tools

taxi

taxi is one repo, four tools for working with the NYC Taxi & Limousine Commission (TLC) trip record data set: a WAF-aware CloudFront downloader (WAF = Web Application Firewall — the layer AWS uses to block traffic that looks like a scraper), a schema-drift analyzer, a normalize step that halts on any data loss unless the operator explicitly acknowledges the drift, and a k6-loadtest rig that drives SQL Server from real parquet or a synthetic generator. The project is MIT-licensed. The downloader script was loosely based on toddwschneider/nyc-taxi-data; the other three tools are original to this repo. See Acknowledgments for details.

• WAF-aware CloudFront downloader with a 5 / 15 / 60 minute exponential backoff ladder and stop-on-local incremental catch-up so nightly crons stay cheap once the mirror is warm.
• Normalizer that treats data loss as a first-class error — missing columns, lossy casts, and silent renames halt the run; explicit ack_date acknowledgment is required before drift is written through.
• K6-based SQL Server load tester with both real-parquet and synthetic-generator modes so you can iterate on a test.js in seconds without waiting for a multi-gigabyte load.

→ Full documentation: https://andrekamman.github.io/taxi-seed/

The site has the deep-dive guides, a cookbook of cross-cutting recipes, an architecture overview, and the design specs; this README stays intentionally short.

Components

Four tools, one repo:

• downloader/ — bash CLI that mirrors the TLC CloudFront bucket to local parquet.
• schema-drift/ — Python CLI that reports column-name and column-shape drift across a mirror.
• normalize/ — Python CLI that rewrites a mirror to a single target schema, refusing to lose data.
• k6-loadtest/ — K6 rig plus a Python preprocessor that turns parquet (or synthetic data) into a self-contained load-test bundle.

Each component has a short README.md that points at the guide on the site; the guide is authoritative.

Quick start

git clone https://github.com/andrekamman/taxi-seed.git
cd taxi
./downloader/download_taxi_data.sh --recent 3 yellow

Downloads ~200 MB in 1–2 minutes on residential broadband. This Quick Start only exercises the downloader (bash + curl — no Python needed). To use the other three tools (schema-drift, normalize, k6-preprocess) you'll additionally need uv sync for the Python side; the Getting Started tutorial walks the full end-to-end path from clone to normalized parquet.

A full-history mirror is roughly 40 GB and takes 6–10 hours end-to-end; the downloader is designed to be resumable, incremental, and cheap to re-run on a schedule rather than something you kick off once and hope survives.

Requirements

• bash 4+, curl (Git for Windows on Windows). Required by the downloader; that's all the downloader needs.
• Python 3.12 or 3.13, uv. Required by the other three tools (schema-drift, normalize, k6-preprocess), which are all Python.
• Disk sized to intent — see the Downloader guide.
• Individual tools list per-guide prerequisites (Go 1.22+ for the K6 binary build, SQL Server for load testing, etc.).

Everything runs on macOS, Linux, and Windows (via Git Bash). CI runs the test suite and the strict docs build on every PR.

Documentation map

The documentation site is the source of truth. High-level sections:

• Getting Started — 10-minute end-to-end walkthrough from clone to normalized parquet.
• Guides — one deep-dive per tool (Downloader, Schema Drift, Normalize, K6 Load Test).
• Cookbook — cross-cutting recipes (nightly cron, DuckDB httpfs querying, corporate proxy, load-testing at scale).
• Architecture — how the four tools fit together end-to-end.
• Reference — configuration keys and exit codes.
• Design Specs — the "why is this shaped this way?" trail.

Contributing

PRs welcome. See docs/contributing.md — also published as the Contributing page on the site — for the dev-setup, test, and PR-checklist details.

License

MIT — see LICENSE.

Acknowledgments

The downloader shell script was loosely based on toddwschneider/nyc-taxi-data (MIT) — specifically its convention for organizing TLC parquet by type and year. The rest of the repo (schema-drift analyzer, normalizer, k6-loadtest rig) is original work and shares no code with Todd's project. See THIRD_PARTY_NOTICES for the full attribution.