data-validator

data-validator validates MIP data-model folders using DuckDB.

Standalone package setup

From the repository root:

cd data-validator/mip-data-validator
poetry install

Command

poetry run data-validator validate-data-model /path/to/data_model_folder

Run as a Python module:

poetry run python -m data_validator validate-data-model /path/to/data_model_folder

Optional threads:

poetry run data-validator validate-data-model /path/to/data_model_folder --threads 8

Collect all errors and emit NDJSON:

poetry run data-validator validate-data-model /path/to/data_model_folder --report-all --format ndjson

Write HTML report to a file:

poetry run data-validator validate-data-model /path/to/data_model_folder --report-all --format html --output report.html

If --format html is used without --output, the report is automatically written under /tmp and the path is printed.

Folder Layout

/path/to/data_model_folder/
  CDEsMetadata.json
  dataset1.csv
  dataset2.csv

Validation Notes

• CSV validation queries files directly with DuckDB and uses fused aggregate checks to reduce scan overhead.
• Folder-level dataset uniqueness is enforced across all CSV files via SQL using normalized codes (trim + lower).