tlf-data-cleaning

Data extraction, transformation, normalization, deduplication, and quality-check pipelines for the TLF ecosystem — the tlf-data-cleaning package described in the TLF-Data-Manager README, built out as a real, installable, tested package.

It solves a concrete problem: government census reports (Bangladesh BBS, Pakistan PBS, etc.) are published as PDFs, not clean CSVs. This package extracts the raw tables out of those PDFs and runs them through a composable set of cleaning rules to produce the canonical schema that tlf-census-stats expects — without hardcoding any of that cleaning logic inside the stats package itself.

Install

pip install -e .
# or, for running the test suite too:
pip install -e ".[dev]"

Modules

Quick example: PDF → clean CSV

from tlf_data_cleaning import (
    CleaningPipeline, RenameColumns, StripWhitespace, CoerceNumeric, DropDuplicates,
)

pipeline = CleaningPipeline([
    RenameColumns({"Division": "region", "District": "subregion",
                   "Total Population": "total_population", "Male": "male",
                   "Female": "female", "Households": "households",
                   "Literacy Rate": "literacy_rate"}),
    StripWhitespace(columns=["region", "subregion"]),
    CoerceNumeric(["total_population", "male", "female", "households", "literacy_rate"]),
    DropDuplicates(subset=["region", "subregion"]),
])

clean_df = pipeline.run_from_pdf("bbs_district_report.pdf", page=12)
print(pipeline.quality_report().summary())

pipeline.run_and_export(clean_df, "data/sample/bangladesh_census_2022.csv")

Canonical schema

tlf-census-stats's CensusLoader expects data in this shape (see its country_profiles.py for exact per-country column aliases):

This package is the upstream step for two of the three ways data reaches tlf-census-stats — it turns a raw government PDF into a CSV in this same shape.

Three ways to get data into tlf-census-stats

1. Already-clean CSV/Excel/JSON — skip this package entirely

If your source is already a clean spreadsheet/CSV/JSON in (or close to) the canonical shape above, just hand it straight to CensusLoader — this package isn't needed at all:

from tlf_census_stats import CensusLoader
df = CensusLoader("already_clean_bangladesh_census.csv", country="bangladesh").load()

2. Flat-country PDF (e.g. Bangladesh, Pakistan) — this package handles the whole thing

Most countries' census PDFs are a flat table per page — extract, clean, and hand off directly:

from tlf_data_cleaning import (
    CleaningPipeline, RenameColumns, StripWhitespace, CoerceNumeric, DropDuplicates,
)
from tlf_census_stats import CensusLoader, StatsReporter

# 1. PDF -> clean CSV (this package)
pipeline = CleaningPipeline([
    RenameColumns({"Division": "region", "District": "subregion",
                   "Total Population": "total_population", "Male": "male",
                   "Female": "female", "Households": "households",
                   "Literacy Rate": "literacy_rate"}),
    StripWhitespace(columns=["region", "subregion"]),
    CoerceNumeric(["total_population", "male", "female", "households", "literacy_rate"]),
    DropDuplicates(subset=["region", "subregion"]),
])
pipeline.run_from_pdf_and_export(
    "bbs_district_report.pdf", page=12,
    output_path="data/sample/bangladesh_census_2022.csv",
)

# 2. clean CSV -> stats report (tlf-census-stats)
reporter = StatsReporter("data/sample/bangladesh_census_2022.csv", country="bangladesh")
reporter.run()

3. India's hierarchical PDF layout — extract here, transform in tlf-census-stats

India's census PDF has a nested INDIA → STATE → DISTRICT → SUB-DISTRICT structure that a generic CleaningPipeline of rename/strip/coerce rules can't reshape — that's what tlf-census-stats's IndiaCensusTransformer is specifically for. This package only does the raw extraction step; the reshaping happens on the other side:

from tlf_data_cleaning import PDFTableExtractor
from tlf_census_stats import IndiaCensusTransformer, CensusLoader, StatsReporter

# 1. Extract raw tables (this package) — no cleaning/reshaping applied
extractor = PDFTableExtractor("india_2011.pdf")
tables = extractor.extract_pages(range(1, 2226), known_header=IndiaCensusTransformer.EXPECTED_HEADER)

# 2. Reshape the hierarchical rows into canonical region/subregion rows (tlf-census-stats)
df = IndiaCensusTransformer().transform(tables)
df.to_csv("data/sample/india_from_pdf.csv", index=False)

# 3. Analyze (tlf-census-stats)
reporter = StatsReporter("data/sample/india_from_pdf.csv", country="india")
reporter.run()

Column names produced by workflow 2's pipeline just need to match the canonical schema table above for a given country's profile — workflow 3 doesn't need column renaming at all, since IndiaCensusTransformer already outputs canonical column names directly.

Known Limitations

• Rotated tables without ruling lines are not yet supported. Some government PDFs embed a wide table as 90°-rotated text so it fits a portrait page (the table visually reads sideways). PDFTableExtractor correctly reconstructs the text of rotated characters (as of 0.1.1), but it still relies on pdfplumber's built-in row/column detection to find the table's shape — and that detection only works reliably when the table has ruling lines around its cells. A rotated table that instead relies on consistent spacing (no visible per-row/per-column borders) will extract with the wrong number of rows/columns even though the text within each cell it does find is now correct. Nepal's census "Table 17: Population by single year of age and sex" is a known example — it's both rotated and unruled, and currently needs to be extracted manually rather than through this package. Properly supporting this layout would require reconstructing table geometry directly from character positions (clustering by x/y position) instead of using pdfplumber's line- or text-based strategies.

Notes on real government PDFs

• pdfplumber's table detection works well on ruled/gridded tables (like the BBS district reports) but can miss tables with no visible borders — pass table_settings to extract_page/extract_all to tune detection (see pdfplumber's docs for vertical_strategy / horizontal_strategy options) if a specific report doesn't extract cleanly.
• Multi-line headers or footnote rows sometimes get pulled in as data rows — inspect extractor.extract_page(n) output before wiring it into a CleaningPipeline, and add a DropRowsWhere rule to filter out any non-data rows if needed.

Tests

python -m pytest tests/ -v

tests/fixtures/generate_fixture.py builds the sample PDF used by the test suite — a small table with thousands separators, stray whitespace, and a duplicate row, mimicking real BBS/PBS-style reports. Regenerate it with:

python tests/fixtures/generate_fixture.py