OSW Sanitizer

osw-sanitizer is a Python package for sanitizing OpenSidewalks (OSW) dataset ZIP files. It is designed to be consumed by the TDEI sanitization service and by other Python workflows that need the same deterministic cleanup behavior.

What It Does

Given a dataset ZIP, the sanitizer runs these passes in order and reports every change it made:

1. Drops files that do not belong — non-OSW filenames and macOS packaging metadata (__MACOSX/, ._* resource forks, .DS_Store).
2. Removes broken tags — JSON null and numeric NaN property values. Look-alike strings ("None", "null", "nan", "n/a", "na") and falsy but meaningful values (0, false, "") are kept.
3. Shortens coordinates to a configurable precision, by rounding (default) or truncating.
4. Splits oversized geometry — lines and polygon rings carrying more vertices than the limit are broken into consecutive parts within it.
5. Creates missing nodes for every _u_id / _v_id / _w_id that no node declares.
6. Enforces unique node _ids — identical repeats dropped, conflicting ones re-ided.
7. Collapses duplicate nodes sharing coordinates and tags, repointing references at the survivor.
8. Verifies the graph is intact, then validates the result with python-osw-validation and bundles everything into osw_data.zip.

Installation

pip install osw-sanitizer

For local development:

python -m pip install -e .
python -m pip install pytest coverage

Quick Start

from osw_sanitizer import OSWSanitization, SanitizationConfig

config = SanitizationConfig(
    coordinate_precision=7,
    coordinate_rounding="round",  # or "truncate"
)

result = OSWSanitization(
    input_path="/path/to/input.zip",
    output_dir="/path/to/output",
    config=config,
).sanitize()

if result.success:
    print(result.updated_dataset_zip)   # osw_data.zip, the published bundle
    print(result.fixes_json)
else:
    print(result.message)               # includes any validator issues

Service-Compatible API

OSWSanitization.sanitize_dataset(...) returns the same information as a dictionary:

from osw_sanitizer import OSWSanitization

result = OSWSanitization.sanitize_dataset(
    input_zip_path="/path/to/input.zip",
    output_dir="/path/to/output",
)

print(result["success"])
print(result["message"])
print(result["updated_dataset_zip"])
print(result["fixes_json"])

SanitizationProcessor is retained as an alias of OSWSanitization.

Configuration

Option	Default	Description
coordinate_precision	7	Maximum decimal places retained for coordinate values.
coordinate_rounding	"round"	How a too-long coordinate is shortened: "round" to the nearest value, halves away from zero, or "truncate" toward zero.
max_geometry_vertices	2000	Lines and polygon rings carrying more vertices than this are split into parts within the limit.
validate_output	True	Validate the sanitized dataset and publish osw_data.zip. Set False to sanitize without judging the result against the OSW schema.

The configuration names match the OSW formatter and validator packages where applicable. Configuration is passed in code — the package reads no environment variables and no .env file.

Each option can also be passed directly to the constructor:

OSWSanitization(input_path=..., output_dir=..., coordinate_precision=6)

Input Requirements

input_path must point to an existing .zip archive. The sanitizer returns an unsuccessful SanitizationResult without writing any output when:

Input	Message
Missing path	Input dataset path is missing
Path does not exist	Input dataset not found at path: <path>
Not a .zip filename	Input dataset must be a .zip file: <path>
.zip filename that is not a zip archive	Input dataset is not a valid zip archive: <path>

Supported Dataset Files

Supported filenames come from python-osw-validation, so the sanitizer keeps exactly the files the OSW validator accepts. The dataset keys are OSW_DATASET_FILES:

• edges
• lines
• nodes
• points
• polygons
• zones

Supported filename forms are:

• <dataset>.geojson
• <dataset>.OSW.geojson
• *.<dataset>.geojson
• *.<dataset>.OSW.geojson

Matching is case-insensitive.

Removed Files

These are omitted from the sanitized output and recorded under removedFiles in fixes.json:

File	fixType
Non-OSW filenames, including unsupported .geojson names and non-geojson files	unsupported_file_removed
__MACOSX/ entries, ._* resource forks, .DS_Store	macos_metadata_removed

Coordinate Precision

Coordinates already within coordinate_precision are left byte for byte as they are — never padded with trailing zeros — and are not reported in precisionUpdates. Only longer fractions are shortened, either way:

Input	"round"	"truncate"
-122.123456789	-122.1234568	-122.1234567
47.12345674	47.1234567	47.1234567
47.12345675	47.1234568	47.1234567
-47.12345675	-47.1234568	-47.1234567

Rounding moves a point by at most half a unit of the last digit and has no directional bias; truncation always moves toward zero, so it biases a dataset slightly. Either way a coordinate can shift, which is why an edge endpoint can end up marginally off its node — see Graph Verification.

Geometry Vertex Limit

Features in edges, lines, polygons, and zones are held to max_geometry_vertices, counted the way the OSW validator counts: every LineString coordinate, and every polygon ring vertex except the repeated closing one. A feature over the limit is split into consecutive parts, each <id>-part-<n>, and logged under splitGeometries.

• Lines are cut into runs, with consecutive parts sharing the vertex they were cut at, so the line stays continuous and no vertex is lost. For edges, each cut becomes a <id>-split-node-<n> and the parts' _u_id / _v_id are rewired through it; the nodes themselves are materialized by the reference pass at the coordinates those ids imply.
• Polygon rings are cut into runs, each closed back on itself into its own ring, each keeping at least three vertices.
• A polygon with interior rings has no split that carries its holes, so it is left as it is and logged as oversized_geometry_not_split.

Parts are sized evenly rather than packed to the limit — 2500 vertices at a limit of 2000 becomes 1251 + 1250, not 2000 + 501. Packing leaves a short tail, and a few vertices off a smooth curve collapse into a degenerate sliver once the coordinates are rounded.

Two things to know about splitting polygons:

• Closing each ring chunk cuts across the polygon, which reproduces the original area only where the ring is convex. A concave ring's parts tile it approximately, so treat this as a way to satisfy the vertex limit rather than an exact partition.
• A limit small enough that a part spans less ground than coordinate_precision resolves will still produce degenerate rings, which the validator then rejects as invalid geometry. This does not arise at the default limit; it needs a limit orders of magnitude smaller on densely sampled curves.

Node Topology

Every _u_id / _v_id (edges) and _w_id (zones) must name a node. The sanitizer treats the reference id as authoritative rather than repointing it:

• _u_id is the edge's first vertex, _v_id its last.
• The n-th _w_id is the n-th vertex of the zone's outer ring, with the repeated closing vertex dropped first.
• A reference no node declares gets a node created with that exact _id, at the coordinate the reference implies. A dataset can therefore come out with more nodes than it went in with.
• References that already resolve are left alone, even when the node sits away from the vertex.
• If the coordinate cannot be determined — a _w_id count that does not match the ring, or an empty reference id — nothing is invented and the reference is logged under unresolvedReferences.

Duplicate Nodes

Duplicate node _ids are not allowed: a reference has to name exactly one node. Within the nodes file the first feature to claim an _id keeps it, so existing references stay pointed at the same node. A later repeat is dropped when identical, and otherwise reassigned the next free <id>-<n>.

Nodes that share both their coordinates and their tags describe the same place, so they are collapsed into the first of them. Every _u_id / _v_id / _w_id pointing at a collapsed node is repointed at the survivor, logged as collapsedNodes and updatedReferences. Comparison happens after rounding, so nodes differing only below the precision limit collapse too. Nodes at the same place with different tags are left alone.

Graph Verification

After the fixes are applied, the sanitizer walks every reference once more and records the outcome under verification in fixes.json:

{
  "verification": {
    "nodeCount": 12,
    "referenceCount": 14,
    "graphIntact": true,
    "danglingReferences": [],
    "misplacedReferences": []
  }
}

• graphIntact is true when every reference resolves to a node that exists.
• danglingReferences holds references the sanitizer already reported as unplaceable; they are findings, not failures, and the run still succeeds.
• misplacedReferences holds references that resolve to a node sitting away from the vertex they describe. The reference is authoritative, so the node is never moved — but rounding can shift an endpoint off its node by up to one unit of the configured precision, and this is where that shows up.
• A dangling reference that was not reported as unplaceable means a preceding pass broke the graph. That fails the run rather than shipping a broken dataset.

Output Artifacts

A validated run publishes osw_data.zip into output_dir, and that is what result.updated_dataset_zip points at. It bundles:

1. The sanitized dataset ZIP, under the same filename as the input ZIP.
2. fixes.json, structured details about every applied change.
3. validation_issues.json, the validator's issues as {"issues": [...]}.

All three are also left loose in output_dir, so result.fixes_json points at fixes.json on disk rather than inside the bundle. The bundle is published whether or not the dataset validates — only success and message differ.

result.updated_dataset_zip   # .../output/osw_data.zip
result.fixes_json            # .../output/fixes.json

With validate_output=False there is no bundle: nothing has vouched for the dataset, so updated_dataset_zip is the sanitized dataset ZIP itself and no validation_issues.json is written.

Output Validation

Once sanitization finishes, the sanitized ZIP is handed to python-osw-validation, configured from the same settings that produced it — coordinate_precision and max_geometry_vertices are passed through, so the output is judged by the limits it was sanitized to.

Either outcome publishes the same three artifacts; what changes is the result and what validation_issues.json holds:

Outcome	success	message	validation_issues.json
Validates	True	what was sanitized	{"issues": []}
Rejected	False	the validator's issues	the issues, so a caller can fix the dataset

A rejected run reports the issues in the message as well:

Sanitized dataset is not a valid OSW dataset.
- edges.geojson (feature 0): "" is shorter than 1 character (at: features[0].properties._u_id)

SanitizedDatasetValidationError keeps the raw issues alongside the rendered messages, and format_issues(...) renders any issue list the same way.

Note that the validator enforces that an edge endpoint sits exactly on its node. Rounding can move an endpoint off its node — the sanitizer reports that under misplacedReferences but does not repair it, so such a dataset sanitizes cleanly and then fails validation here.

fixes.json

Per-file entries carry only the keys that apply:

Key	Written when
removedTags	a null or NaN tag was dropped
precisionUpdates	a coordinate was rounded or truncated
addedNodeReferences	a dangling reference caused a node to be created
unresolvedReferences	a reference could not be placed
addedNodes	nodes were added to the nodes file
removedNodes	a repeated _id on an identical node was dropped
reassignedNodeIds	a repeated _id on a differing node was re-ided
collapsedNodes	duplicate nodes were collapsed
updatedReferences	a reference followed a collapsed node
splitGeometries	a feature exceeded the vertex limit

Alongside them, removedFiles lists dropped files and verification reports the graph check.

{
  "jobId": "",
  "files": [
    {
      "filename": "edges.geojson",
      "removedTags": [
        {
          "featureIndex": 0,
          "tag": "width",
          "value": null
        }
      ],
      "precisionUpdates": [
        {
          "featureIndex": 0,
          "coordinatePath": "coordinates[0]",
          "original": "-122.123456789",
          "updated": "-122.1234568",
          "precision": 7,
          "rounding": "round"
        }
      ]
    }
  ],
  "removedFiles": [],
  "verification": {
    "nodeCount": 0,
    "referenceCount": 0,
    "graphIntact": true,
    "danglingReferences": [],
    "misplacedReferences": []
  }
}

A removed NaN is logged as the string "NaN", so fixes.json stays parseable by strict JSON readers.

Testing

Install the package and test dependencies:

python -m pip install -e .
python -m pip install pytest coverage

Run the unit tests:

python -m pytest

Run the unit tests with coverage enforcement:

coverage run -m pytest
coverage report --fail-under=90

The GitHub Actions unit test workflow writes timestamped test and coverage logs into test_results/ and uploads them to Azure Blob Storage using the AZURE_STORAGE_CONNECTION_STRING secret.

Package metadata is defined in pyproject.toml. setup.py is retained as a compatibility shim for legacy packaging workflows.

Test Datasets

Sample dataset ZIPs are checked in under tests/assets. The six-file OSW datasets (edges, lines, nodes, points, polygons, zones) are generated by tests/dataset_builder.py and carry the OSW 0.3 $schema:

Dataset	Covers	Sanitize result
passed.zip	clean dataset, no fixes applied	passes validation
missing_references.zip	dangling _u_id / _v_id / _w_id	passes validation
precision_and_duplicates.zip	over-long coordinates and repeated node ids	passes validation
collapsible_nodes.zip	duplicate nodes that collapse into one	passes validation
rounding_modes.zip	coordinates where round and truncate differ	passes validation
null_and_nan_tags.zip	null / NaN tags and their look-alikes	passes validation
cleanup.zip	macOS metadata and unsupported filenames	passes validation
misplaced_references.zip	references resolving away from their vertex	fails validation
unresolvable_references.zip	references the sanitizer will not guess at	fails validation
failure.zip	non-finite coordinates and property values	fails sanitization
not_a_zip.geojson, corrupt.zip	invalid inputs for the ZIP-only check	rejected as input

The last four fail by design. The two that fail validation still publish osw_data.zip, with the issues in validation_issues.json; pass validate_output=False to skip the gate entirely.

Regenerate them with:

python tests/dataset_builder.py

Single-purpose zips, hand-maintained and not schema-valid, used to exercise individual passes with validate_output=False:

• precision_and_null_tags.zip
• zero_length_edge.zip
• unsupported_files.zip
• nested_dataset.zip

Release Pipelines

GitHub Actions includes package publishing workflows:

• .github/workflows/deploy_to_test.yml publishes to TestPyPI from develop.
• .github/workflows/publish_to_pypi.yml publishes to PyPI from semver tags or manual dispatch.

Both workflows build the package from pyproject.toml and use PYPI_API_TOKEN for authentication.