nzi-pipeline 0.1.0

Sector-agnostic energy modelling pipeline for India's NZI scenario, backed by DuckDB. Local runs + S3-versioned seeds/outputs.

NZI — Net Zero India

A sector-agnostic energy-demand modelling pipeline for India's net-zero pathway analysis. Backed by DuckDB, it ingests Excel inputs, computes PIER parameters, and reproduces RUMI-format demand outputs end-to-end — with a web UI, snapshots, undo, tests, CI, and a one-command Docker deployment.

---

Highlights

• Two sectors live, one framework — residential and transport share the same CLI, schema, and step runner.
• DuckDB-backed pipeline — replaces brittle Excel chains with an auditable, queryable database (residential.duckdb ~3.8 GB, transport.duckdb smaller).
• PIER + RUMI-compatible output — nzi_pipeline writes PIER parameter CSVs, pier_db reimplements the RUMI demand formula and produces byte-comparable demand CSVs.
• FastAPI web UI — browse tables, edit cells, run steps with live log streaming, manage snapshots, and undo any edit.
• Reproducible runs — named DB snapshots, append-only audit log, staleness tracking via dependency graph.
• Tested + CI — 50 pytest cases (33 unit + 17 integration), golden fixtures regenerable on demand, GitHub Actions on every PR.
• Containerised — docker compose up builds the env, seeds both DBs, and serves the UI on port 8000.

---

Quick Start

Local (recommended for development)

# 1. Install
pip install -e ".[dev]"

# 2. Seed database from Excel workbooks (~4 min, includes 12 GB climate data)
python -m nzi_pipeline seed --sector residential
python -m nzi_pipeline seed --sector transport

# 3. Run the parameter pipeline
python -m nzi_pipeline run --sector residential
python -m nzi_pipeline run --sector transport

# 4. Compute demand (RUMI-format output)
python -m pier_db run --sector D_RES
python -m pier_db run --sector D_TRANS

# 5. (Optional) Start the web UI
python -m frontend            # http://127.0.0.1:8000/

Docker

docker compose up -d --build
docker compose logs -f        # watch first-boot seed
# UI on http://<host>:8000/

Full EC2 deployment guide: DEPLOY.md.

Legacy Excel pipeline (reference only)

powershell -ExecutionPolicy Bypass -File setup.ps1
cd General\Residential_Sector_Data\Residential_Workflow_FromInput
python run_all.py

---

Architecture

                  Excel Workbooks (RUMI_Input/)
                              |
                       nzi_pipeline seed
                              |
                              v
                  residential.duckdb / transport.duckdb
                       /         |         \
              nzi_pipeline    frontend     pier_db
                  run         (web UI)      run
                   |             |            |
                   v             v            v
            PIER Parameter   Edit / Run    Demand Output
              CSVs (~34)     / Snapshot    CSVs (~57)
                              / Undo

Component summary

Component	What it does
nzi_pipeline/	Sector-agnostic pipeline framework — seed Excel into DuckDB, compute parameters (ES_Demand from 112M climate rows, SEC, stock-flow), export PIER CSVs.
pier_db/	Reimplements RUMI's demand formula (NC × NI × ES_Demand × UP × TSR × ELS × SEC) reading from DuckDB instead of CSV. Output is RUMI-format.
frontend/	FastAPI single-page UI — browse, edit, run, snapshot, undo. Same code paths as the CLI, so all UI actions appear in the audit log.
tests/	pytest harness — unit + integration + fixture-based regression with golden manifests.
scripts/	Shell wrappers used by both devs and CI (test.sh, seed.sh, run.sh, dev-docker.sh).

---

CLI Reference

nzi_pipeline — parameter pipeline

python -m nzi_pipeline seed   --sector <name>           # Excel -> DuckDB
python -m nzi_pipeline seed   --sector <name> --force   # drop + re-seed
python -m nzi_pipeline run    --sector <name>           # run all steps
python -m nzi_pipeline run    --sector <name> --force   # force rebuild
python -m nzi_pipeline steps  --sector <name>           # list steps

Implemented sectors: residential, transport.

pier_db — demand computation

python -m pier_db run --sector D_RES
python -m pier_db run --sector D_TRANS
python -m pier_db run --sector D_RES --output ./output  # custom output dir

frontend — web UI

python -m frontend                          # 127.0.0.1:8000
python -m frontend --reload                 # dev auto-reload
python -m frontend --port 9000 --no-browser

---

Frontend Features

A single-page UI on top of nzi_pipeline.core.*. See frontend/README.md for full endpoint reference.

Tab	What it does
Dashboard	Staleness summary, recent edits, snapshots at a glance.
DAG	Visual step graph (left → right by topology); click for upstream/downstream lineage.
Tables	Paginated browser for input / assumption / computed / _meta; in-cell edits to input and assumption are logged.
Run	Plan, run-stale, or force rebuild — with live log streaming via SSE.
Audit & Undo	Append-only edit history; one click reverts any edit.
Snapshots	Create, restore, or delete named DB snapshots.

DuckDB doesn't allow concurrent rw connections to one file, so the frontend caches a per-pipeline connection and releases it after each job so the CLI / pier_db can run between jobs.

---

Repository Structure

NZI/
├── nzi_pipeline/                      # DuckDB pipeline framework
│   ├── core/                          #   Generic: db, config, registry, refresh, export
│   └── sectors/
│       ├── residential/               #   Residential: seed, defaults, schema, 7 steps
│       └── transport/                 #   Transport: seed, defaults, schema, steps
├── pier_db/                           # Demand computation (replaces rumi_demand)
│   ├── config/                        #   RUMI mapping CSVs (Common, Demand)
│   ├── demand.py                      #   Core formula: NC x UP x NI x TSR x ELS x SEC x ES
│   ├── parameters.py                  #   Read parameters from DuckDB
│   └── export.py                      #   Write RUMI-format output CSVs
├── frontend/                          # FastAPI web UI
│   ├── app.py                         #   REST + SSE endpoints
│   ├── __main__.py                    #   `python -m frontend` entry point
│   └── static/                        #   index.html + app.js + style.css (no build)
├── tests/                             # pytest suite (50 tests)
│   ├── unit/                          #   33 tests, in-memory DuckDB, <1s
│   ├── integration/                   #   17 tests, ~55s, fixture-driven
│   └── fixtures/{residential,transport}_min/  # 1.5 MB sliced Parquet inputs
├── scripts/                           # dev + CI shell wrappers
│   ├── test.sh                        #   lint + unit + integration
│   ├── seed.sh / run.sh               #   convenience wrappers
│   ├── build_{residential,transport}_fixture.py
│   └── build_manifest.py              #   regenerate golden output hashes
├── .github/workflows/ci.yml           # lint + unit + integration + docker-build
├── General/Residential_Sector_Data/   # Professor's reference data + legacy pipeline
├── PIER/                              # Professor's PIER scenario outputs (read-only)
├── Dockerfile, docker-compose.yml,
│   docker-entrypoint.sh, environment.yml
├── DEPLOY.md                          # EC2 deployment guide
├── aws_deployment_plan.csv            # Spot-Batch + S3 production plan
├── pyproject.toml                     # Package definition (MIT license)
├── residential.duckdb / transport.duckdb   # Generated, git-ignored
└── README.md

---

Testing

bash scripts/test.sh all          # lint + unit + integration (~55s)
bash scripts/test.sh unit         # 33 tests, <1s
bash scripts/test.sh integration  # 17 tests, ~55s

Layer	Coverage
Unit	In-memory DuckDB; covers edit, snapshot, _quote, pier_db static maps.
Integration	Database lifecycle, real-DB smoke (auto-skips if locked), full residential + transport pipeline e2e on tiny fixtures.
Manifest check	Golden-output regression at 1e-6 relative tolerance.

Regenerating fixtures / golden manifests

python scripts/build_residential_fixture.py
python scripts/build_transport_fixture.py
python scripts/build_manifest.py residential   # after intentional math change
python scripts/build_manifest.py transport

CI

.github/workflows/ci.yml runs on every PR to main / op_dev:

1. Lint — ruff strict on tests/, advisory on nzi_pipeline/ + pier_db/.
2. Unit tests — Python 3.11, pip-cached.
3. Integration tests — fixture-driven; real-DB tests auto-skip in CI.
4. Docker build — only on push to main; uses GHA cache.

In-flight runs on the same branch are auto-cancelled to save minutes.

---

Validation against the professor's reference

cd General/Residential_Sector_Data/Residential_Workflow_FromInput
python validate_parameters_keyed.py    # PIER params vs professor
python validate_demand_keyed.py        # demand output vs professor
python compare_outputs.py              # pier_db output vs professor (latest)

Current residential match rates:

• Parameters: 32 / 34 files IDENTICAL.
• Demand: 37 / 56 files within 1% (after year fix); 13 files >10% remain in cooling + cook fuel mix (tracked in compare_outputs_report.md).

---

Adding a New Sector

The framework is designed so new sectors plug in cleanly:

1. Create nzi_pipeline/sectors/<sector>/ with config.py, defaults.py, schema.sql, seed.py.
2. Add computation steps under steps/.
3. Register the sector in nzi_pipeline/cli.py.
4. Create pier_db/config/Demand/<D_SECTOR>/ with mapping CSVs.
5. Add a fixture builder under scripts/ and a golden manifest.

Transport is the live example — copy its layout.

---

Documentation

Document	Covers
TABLES.md	Database schemas (input / assumption / computed / _meta) and what every table stores
DEPLOY.md	EC2 deployment via Docker Compose
aws_deployment_plan.csv	Production plan (Spot-Batch, S3, Step Functions)
frontend/README.md	Full UI tour and API reference
General/Residential_Sector_Data/README_Residential.md	Pipeline overview, inputs, outputs, dependencies
General/Residential_Sector_Data/CODE_STRUCTURE.md	Scripts, modules, step-by-step data flow
General/Residential_Sector_Data/DB_PROPOSAL.md	Why DuckDB, storage model, migration plan
General/Residential_Sector_Data/DB_SCHEMA.md	Table and view definitions
General/Residential_Sector_Data/DB_REFRESH.md	Dependency graph and refresh() function

---

Project Conventions

• Never touch professor folders — General/Residential_Sector_Data/Parameters/ and PIER/Scenarios/ are read-only references.
• Generate static files in the pipeline — don't copy from the professor; produce them in s06_export.py.
• Match RUMI semantics exactly — pier_db mirrors github.com/prayas-energy/Rumi.
• Lint — strict for tests/; existing nzi_pipeline/ + pier_db/ warnings are advisory until cleanup.

---

Git Notes

• residential.duckdb / transport.duckdb are git-ignored — regenerate with python -m nzi_pipeline seed.
• PIER/ is git-ignored (managed by external team).
• RUMI_Input_code/ is git-ignored (legacy-pipeline output).
• index.lock errors — VS Code and terminal can race; run rm .git/index.lock (or clear-git-lock.ps1 on Windows).
• DuckDB locked — if the frontend crashed, Stop-Process -Id <pid> -Force, then restart.

---

License

MIT — see pyproject.toml.