utility-network-schema-qa-toolkit 1.0.0

License-free validation for Utility Network schemas, mappings, rules, and migration QA artifacts

Utility Network Schema QA Toolkit

utility-network-schema-qa-toolkit V1.0 is an open-source, license-free Python toolkit for validating exported Utility Network schemas, source-to-target mappings, per-mapping Definition Queries, domains, asset classifications, declared rules, Data Reference inventories, dirty-area exports, and migration QA reports.

It works with CSV, TSV, JSON, YAML, XLSX, and XLSM artifacts without ArcPy or a proprietary GIS runtime. It does not connect to or modify a live Utility Network.

What V1.0 provides

• Strict project manifests and immutable canonical data models.
• Header aliases for common schema, mapping, and Data Reference workbook exports.
• Safe, non-executing analysis of a documented SQL-style filter subset.
• Source/target dataset, field, geometry, required-field, type, and transformation QA.
• Domain, coded-value, explicit crosswalk, asset group, and asset type QA.
• Explainable YAML civil-engineering classification rules with deterministic priority.
• Data Reference, attribute-rule dependency, network-rule, terminal, and dirty-area checks.
• Stable Finding codes, exact-code severity overrides, deterministic sorting, and SHA-256 fingerprints.
• JSON, CSV, Markdown, and self-contained accessible HTML reports.
• A Python API and the un-schema-qa command-line interface.
• Complete synthetic water, wastewater, and stormwater projects.

The toolkit reviews exported metadata and declared relationships. It does not execute SQL, Arcade, or Python from inputs; perform a data load; build topology; trace a network; or replace engineering review.

Installation

Python 3.10 or newer is required. After the V1 package is published to the package index:

pip install utility-network-schema-qa-toolkit

To install the current repository checkout:

python -m pip install .

Verify the installation:

un-schema-qa version
un-schema-qa list-checks

Quick start

Create a starter project:

un-schema-qa init my-water-qa --profile water
un-schema-qa validate my-water-qa/project.yml

Run a complete repository example:

un-schema-qa validate examples/water/project.yml
un-schema-qa validate examples/wastewater/project.yml
un-schema-qa validate examples/stormwater/project.yml

Reports are written to the manifest's output directory. Override the destination or formats when needed:

un-schema-qa validate examples/water/project.yml \
  --output review-reports \
  --format json \
  --format html

Exit codes are 0 for pass (and warnings by default), 1 for warnings with --fail-on-warning, 2 for validation failure, and 3 for configuration/input/runtime errors.

Project manifest

Only source schema, target schema, and mappings are required. Other inventories activate additional checks.

project:
  name: water-migration
  profile: water
inputs:
  source_schema: data/source_schema.csv
  target_schema: data/target_schema.csv
  mappings: data/mappings.csv
  source_domains: data/source_domains.csv
  target_domains: data/target_domains.csv
  asset_types: data/asset_types.csv
  data_reference: data/data_reference.csv
  attribute_rules: data/attribute_rules.csv
  network_rules: data/network_rules.csv
  terminals: data/terminals.csv
  dirty_areas: data/dirty_areas.csv
  engineering_rules: rules/engineering_rules.yml
validation:
  enabled: [schema, mapping, filters, domains, asset_classification]
  fail_on: error
  severity_overrides:
    FILTER_EXPECTED_COUNT_MISSING: info
outputs:
  directory: reports
  formats: [json, csv, markdown, html]

Relative paths resolve from project.yml. Unknown manifest keys, checks, profiles, severities, and formats fail early.

Per-mapping Definition Queries and filters

Every source-to-target mapping may carry its own source-side SQL-style filter. This lets one source dataset route different record populations to different target datasets, asset groups, or asset types.

Source	Target classification	Definition Query	Purpose
LegacyWaterLine	WaterLine / Transmission Main	network_role = 'Transmission' AND lifecycle_status = 'Active'	Select active transmission mains.
LegacyWaterLine	WaterLine / Distribution Main	network_role = 'Distribution' AND lifecycle_status = 'Active'	Select active distribution mains.
LegacySewerLine	WastewaterLine / Gravity Main	flow_regime = 'Gravity' AND lifecycle_status <> 'Abandoned'	Select in-scope gravity mains.

Each mapping can also record purpose, expected_count, selected_count, and loaded_count. The filter validator checks:

• supported syntax and balanced expressions;
• fields referenced by the source-side filter;
• missing purpose or expected scope;
• recorded empty selections;
• expected-versus-selected and selected-versus-loaded count differences;
• missing filters when one source is split into multiple mappings; and
• possible overlap for simple equality/IN partitions.

Overlap analysis is deliberately conservative. It can prove simple partitions disjoint or flag possible overlap; complex expressions return unknown rather than an unsafe guarantee. Datastore-specific SQL semantics remain the responsibility of the project team. The toolkit does not define a target-side Definition Query feature.

Civil-engineering-informed asset classification

Mapping by similar names alone is often insufficient. V1.0 can evaluate explicit YAML rules against engineering context and report the rule ID, target classification, explanation, missing attributes, conflicts, and disagreement with the selected mapping.

Useful context includes:

Context	Review use
Facility function and network role	Transmission, distribution, collection, service, storage, or control.
Hydraulic operation	Pressurized, gravity, or open-channel behavior.
Geometry and structure type	Pipe, device, culvert, channel, inlet, junction, or other structure form.
Material, lining, and coating	Construction and rehabilitation distinctions.
Diameter, width, height, and capacity	Scale, section, and design function.
Pressure class, slope, invert, and flow direction	Operating and conveyance behavior.
Installation and lifecycle context	Installation method/date and current asset state.

Conceptual examples:

• Water transmission versus distribution mains can use network role, pressure class, nominal diameter, and lifecycle status.
• Wastewater gravity mains can use gravity regime, positive slope, and upstream/downstream inverts; force mains can use pressurized pump-discharge operation.
• Stormwater classification can distinguish closed pipes, culverts, open channels, and point structures using geometry, structure form, and cross-section dimensions.

Example rule:

rules:
  - rule_id: wastewater-gravity-main
    priority: 10
    conditions:
      - field: flow_regime
        operator: equals
        value: gravity
      - field: slope
        operator: greater_than
        value: 0
    target_asset_group: Main
    target_asset_type: Gravity Main
    explanation: Gravity operation with positive slope supports gravity conveyance.

Allowed operators are equals, not_equals, in, not_in, greater_than, greater_or_equal, less_than, less_or_equal, exists, and contains. Rules support all or any condition mode and never change a mapping. Engineering attributes support and document a decision; they do not replace target domains, Utility Network rules, design standards, or review by qualified GIS and engineering staff.

Python API

from pathlib import Path
from un_schema_qa import load_project, validate_project, write_reports

project = load_project(Path("project.yml"))
result = validate_project(project)
paths = write_reports(result, formats=("json", "html"))

ValidationResult includes project name, toolkit/schema versions, fingerprint, validators actually run, sorted findings, counts, and aggregate status.

Examples

• Water example: transmission/distribution filters, engineering rule matches, Data Reference partitions, and dirty-area grouping.
• Wastewater example: gravity/force-main rules, slope/inverts, pump terminal, and a deliberate rule gap.
• Stormwater example: pipes, culverts, open channels, structures, and two dirty-area groups.

All example names, counts, identifiers, and infrastructure records are synthetic.

Documentation

• Architecture
• Configuration
• Input formats and aliases
• Validation rules
• Finding code catalog
• Python API
• CLI
• Report schema
• Water tutorial
• Wastewater tutorial
• Stormwater tutorial

Development and governance

• Changelog
• Contributing
• Code of Conduct
• Security Policy

Quality commands:

ruff format --check .
ruff check .
mypy src
pytest

Security

Do not commit credentials, tokens, secured service URLs, connection strings, authentication profiles, restricted schema workbooks, or sensitive infrastructure data. Treat terminal logs and generated reports as review artifacts: inspect them before sharing and store them according to their data classification.

The toolkit performs no network requests and does not evaluate input SQL, Arcade, Python, or template expressions.

License

Released under the MIT License.