tlf-data-cleaning

Data extraction, transformation, normalization, deduplication, and quality-check pipelines for the TLF ecosystem — the tlf-data-cleaning package described in the TLF-Data-Manager README, built out as a real, installable, tested package.

It solves a concrete problem: government census reports (Bangladesh BBS, Pakistan PBS, etc.) are published as PDFs, not clean CSVs. This package extracts the raw tables out of those PDFs and runs them through a composable set of cleaning rules to produce the canonical schema that tlf-census-stats expects — without hardcoding any of that cleaning logic inside the stats package itself.

Install

pip install -e .
# or, for running the test suite too:
pip install -e ".[dev]"

Modules

Quick example: PDF → clean CSV

from tlf_data_cleaning import (
    CleaningPipeline, RenameColumns, StripWhitespace, CoerceNumeric, DropDuplicates,
)

pipeline = CleaningPipeline([
    RenameColumns({"Division": "region", "District": "subregion",
                   "Total Population": "total_population", "Male": "male",
                   "Female": "female", "Households": "households",
                   "Literacy Rate": "literacy_rate"}),
    StripWhitespace(columns=["region", "subregion"]),
    CoerceNumeric(["total_population", "male", "female", "households", "literacy_rate"]),
    DropDuplicates(subset=["region", "subregion"]),
])

clean_df = pipeline.run_from_pdf("bbs_district_report.pdf", page=12)
print(pipeline.quality_report().summary())

pipeline.run_and_export(clean_df, "data/sample/bangladesh_census_2022.csv")

Canonical schema

tlf-census-stats's CensusLoader expects data in this shape (see its country_profiles.py for exact per-country column aliases):

This package is the upstream step for two of the three ways data reaches tlf-census-stats — it turns a raw government PDF into a CSV in this same shape.

Three ways to get data into tlf-census-stats

1. Already-clean CSV/Excel/JSON — skip this package entirely

If your source is already a clean spreadsheet/CSV/JSON in (or close to) the canonical shape above, just hand it straight to CensusLoader — this package isn't needed at all:

from tlf_census_stats import CensusLoader
df = CensusLoader("already_clean_bangladesh_census.csv", country="bangladesh").load()

2. Flat-country PDF (e.g. Bangladesh, Pakistan) — this package handles the whole thing

Most countries' census PDFs are a flat table per page — extract, clean, and hand off directly:

from tlf_data_cleaning import (
    CleaningPipeline, RenameColumns, StripWhitespace, CoerceNumeric, DropDuplicates,
)
from tlf_census_stats import CensusLoader, StatsReporter

# 1. PDF -> clean CSV (this package)
pipeline = CleaningPipeline([
    RenameColumns({"Division": "region", "District": "subregion",
                   "Total Population": "total_population", "Male": "male",
                   "Female": "female", "Households": "households",
                   "Literacy Rate": "literacy_rate"}),
    StripWhitespace(columns=["region", "subregion"]),
    CoerceNumeric(["total_population", "male", "female", "households", "literacy_rate"]),
    DropDuplicates(subset=["region", "subregion"]),
])
pipeline.run_from_pdf_and_export(
    "bbs_district_report.pdf", page=12,
    output_path="data/sample/bangladesh_census_2022.csv",
)

# 2. clean CSV -> stats report (tlf-census-stats)
reporter = StatsReporter("data/sample/bangladesh_census_2022.csv", country="bangladesh")
reporter.run()

3. India's hierarchical PDF layout — extract here, transform in tlf-census-stats

India's census PDF has a nested INDIA → STATE → DISTRICT → SUB-DISTRICT structure that a generic CleaningPipeline of rename/strip/coerce rules can't reshape — that's what tlf-census-stats's IndiaCensusTransformer is specifically for. This package only does the raw extraction step; the reshaping happens on the other side:

from tlf_data_cleaning import PDFTableExtractor
from tlf_census_stats import IndiaCensusTransformer, CensusLoader, StatsReporter

# 1. Extract raw tables (this package) — no cleaning/reshaping applied
extractor = PDFTableExtractor("india_2011.pdf")
tables = extractor.extract_pages(range(1, 2226), known_header=IndiaCensusTransformer.EXPECTED_HEADER)

# 2. Reshape the hierarchical rows into canonical region/subregion rows (tlf-census-stats)
df = IndiaCensusTransformer().transform(tables)
df.to_csv("data/sample/india_from_pdf.csv", index=False)

# 3. Analyze (tlf-census-stats)
reporter = StatsReporter("data/sample/india_from_pdf.csv", country="india")
reporter.run()

Column names produced by workflow 2's pipeline just need to match the canonical schema table above for a given country's profile — workflow 3 doesn't need column renaming at all, since IndiaCensusTransformer already outputs canonical column names directly.

Known Limitations

• Rotated tables without ruling lines are not yet supported. Some government PDFs embed a wide table as 90°-rotated text so it fits a portrait page (the table visually reads sideways). PDFTableExtractor correctly reconstructs the text of rotated characters (as of 0.1.1), but it still relies on pdfplumber's built-in row/column detection to find the table's shape — and that detection only works reliably when the table has ruling lines around its cells. A rotated table that instead relies on consistent spacing (no visible per-row/per-column borders) will extract with the wrong number of rows/columns even though the text within each cell it does find is now correct. Nepal's census "Table 17: Population by single year of age and sex" is a known example — it's both rotated and unruled, and currently needs to be extracted manually rather than through this package. Properly supporting this layout would require reconstructing table geometry directly from character positions (clustering by x/y position) instead of using pdfplumber's line- or text-based strategies.

• Sparsely-lined, non-rotated tables can also extract with incorrect shape and truncated cell text. This isn't limited to rotated content — the same underlying issue (a table with few or no per-row ruling lines, relying on spacing alone) affects ordinary upright tables too. On Nepal's "Table 01: Number of households by type of ownership," using table_settings={"vertical_strategy": "text", "horizontal_strategy": "text"} (needed since the table lacks per-row lines) produces a much closer row count than pdfplumber's lines-strategy default, but individual cells' leading characters get silently clipped (e.g. "Nepal" extracts as "epal", "Urban/Rural" as "rban/Rural") — pdfplumber's computed column boundary lands in the middle of the first character's glyph rather than safely before it. This was root-caused by comparing the detected cell boundary against actual character positions, but is not fixable via table_settings tuning (snap_x_tolerance, join_x_tolerance, text_x_tolerance were all tried with no effect on the output). A real fix would need custom per-cell text reconstruction with boundary padding — attempted once, but reverted after it introduced a regression on cleanly-gridded tables (specifically, long header text overflowing its detected column width got truncated at both ends instead of one). Extract tables like this manually for now, the same as the rotated-and-unruled case above.

Notes on real government PDFs

• pdfplumber's table detection works well on ruled/gridded tables (like the BBS district reports) but can miss tables with no visible borders — pass table_settings to extract_page/extract_all to tune detection (see pdfplumber's docs for vertical_strategy / horizontal_strategy options) if a specific report doesn't extract cleanly.
• Multi-line headers or footnote rows sometimes get pulled in as data rows — inspect extractor.extract_page(n) output before wiring it into a CleaningPipeline, and add a DropRowsWhere rule to filter out any non-data rows if needed.

Tests

python -m pytest tests/ -v

tests/fixtures/generate_fixture.py builds the sample PDF used by the test suite — a small table with thousands separators, stray whitespace, and a duplicate row, mimicking real BBS/PBS-style reports. Regenerate it with:

python tests/fixtures/generate_fixture.py