tlf-geo-auto

Auto-detect and resolve geographic columns (province, district, local level) in messy CSV, XLSX, or JSON datasets — no manual column configuration required.

Part of TLF (The Living Fact), Corpola Tech's civic-data interoperability toolkit for Nepal. Built on top of tlf-geo's official NSO administrative registry and tlf-core's header-resolution engine.

The problem

tlf-geo can resolve a Nepali place name to its official administrative code — but you have to tell it exactly which column holds the district, which holds the local level, and so on. That's fine for a dataset you built yourself, but real-world survey exports rarely cooperate:

• Column headers vary wildly (जिल्ला, dist, District_Name, or nothing recognizable at all)
• Place names have typos, inconsistent spelling, or missing suffixes
• The same local-level name can exist in multiple districts
• A row can have the right place name paired with the wrong district

tlf-geo-auto handles all of this automatically, and — critically — never silently guesses. Anything it isn't confident about gets surfaced to you for a quick decision, never quietly miscategorized or dropped.

Installation

pip install tlf-geo-auto

Requires tlf-geo and tlf-core (installed automatically as dependencies).

Quick start

tlf-geo-auto resolve survey.csv --output survey_clean.csv

That's it. The tool will:

1. Load your file (CSV, XLSX, or JSON — detected automatically by extension)
2. Drop any fully-empty rows
3. Detect which columns contain province/district/local-level data
4. Resolve every clean match automatically
5. Walk you through anything ambiguous, misspelled, or inconsistent
6. Save a cleaned file with official NSO codes added

Example session

$ tlf-geo-auto resolve survey.csv --output clean.csv

Loaded 300 rows from survey.csv
Dropped 9 fully-empty row(s).
Detected district: column 'जिल्ला' (header, confidence 0.95)
Detected local_level: column 'स्थानीय तह' (content, confidence 0.76)

Reviewing 12 issue(s) in 'स्थानीय तह'

The following local level values could not be matched with full confidence.
For each one, pick the correct place from the list, or enter 0 to leave it
unresolved for now.
──────────────────────────── Tikpaur (not_found) ─────────────────────────────
District on file: kailali
Affects 1 row(s): [8]
┏━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┓
┃ # ┃ Matched On ┃ Official Name ┃ Level        ┃ District ┃ Code  ┃
┡━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━┩
│ 1 │ tikapur    │ Tikapur       │ municipality │ kailali  │ 77113 │
└───┴────────────┴───────────────┴──────────────┴──────────┴───────┘
Which one is correct for 'Tikpaur'? (0 = none of these / skip) [0/1]: 1

Saved cleaned dataset to clean.csv

Output

tlf-geo-auto augments your file, it doesn't replace it. Every original column stays intact. For each detected geo level, it adds:

New column	Meaning
{level}_code	The official NSO administrative code (e.g. district_code, local_level_code)
{level}_status	resolved, resolved_by_review, ambiguous, not_found, or invalid_district
{level}_error_reason	Human-readable explanation for anything not cleanly resolved

If you confirm an answer during review, the tool also corrects the original text column to match — since at that point it's a human-confirmed fact, not a guess. Rows that auto-resolve without any review are never touched beyond having code columns added; their original text is left exactly as given.

How detection works

For each column, tlf-geo-auto tries two signals, in order:

1. Header match (fast) — checks if the column name matches a known alias (via tlf-core's FieldResolver) for district, province, or municipality.
2. Content match (fallback) — if the header is missing, misspelled, or unrecognized, samples the column's actual values and checks whether they resolve as real places (via tlf-geo's GeoResolver.search()). Only counts as strong evidence at exact/near-exact match rates.

This means a column can be detected correctly even with a header tlf-core has never seen before, as long as its actual values are real place names.

How review works

Every row that can't be cleanly auto-resolved falls into one of these categories:

• ambiguous — the name matches multiple real places even after considering district context; pick the correct one from a list.
• not_found — no exact match; shown up to 3 fuzzy-matched suggestions only if genuinely similar (a similarity threshold filters out coincidental near-matches that aren't real typos).
• invalid_district — the place name is real, but doesn't exist in the district your row claims; shown where it actually exists instead, so you can confirm and correct the district.

Grouping: identical problems are grouped and reviewed once, regardless of how many rows share them — e.g. 50 rows with the same typo become one prompt, not 50. Rows are only grouped together when the answer would genuinely be the same for all of them (a not_found name is grouped regardless of district, since district doesn't change that outcome; ambiguous/invalid-district cases are kept separate per district, since the correct answer can differ).

Never a silent auto-approval: no confidence threshold auto-accepts a fuzzy match, no matter how close. Every non-exact match is shown to a human before being written to the file. This is a deliberate design choice — "never guess, never drop" applies to every layer of this tool, not just the underlying registry.

CLI options

tlf-geo-auto resolve INPUT_PATH --output OUTPUT_PATH [OPTIONS]

  --output, -o TEXT           Path to write the cleaned file. [required]
  --skip-empty-rows /
  --keep-empty-rows           Drop fully-blank rows before processing.
                               (default: skip)
  --no-review                 Skip interactive review entirely; ambiguous/
                               unresolved rows are left as-is with their
                               status columns intact for later inspection.

Supported input/output formats: .csv, .xlsx, .xls, .json (auto-detected by file extension).

Design philosophy

tlf-geo-auto follows the same principle as every package in the TLF ecosystem:

Never guess, never drop. Unresolved or uncertain data always surfaces with enough context to fix it by hand — it's never silently miscategorized, and it's never silently discarded.

Concretely, this means:

• No confidence threshold ever auto-approves a non-exact match
• A correction only ever happens after explicit human confirmation
• Everything left unresolved stays visibly flagged in the output, never dropped from the dataset

Known limitations

• No batch/non-interactive review mode yet — --no-review skips review entirely rather than applying an automated policy; there's currently no middle ground like "auto-apply the top suggestion above score X."
• No standalone audit report — a JSON/HTML summary of everything reviewed and changed is planned but not yet built; for now, the {level}_status and {level}_error_reason columns in the output file are the audit trail.
• Assumes NSO/CBS-standard administrative codes — if your dataset already contains codes from a different registry (e.g. postal codes), this tool doesn't detect or account for that.

Related packages

• tlf-core — header/value normalization registry
• tlf-geo — the underlying Nepal administrative boundary resolver
• tlf-pgreconcile — normalization-aware Postgres database reconciliation

License

MIT