UKAM OS Builder

Build OS address data for uk_address_matcher from either NGD (National Geographic Database) or ABP (AddressBase Premium).

Requirements

• Python 3.10+
• OS Data Hub package and version IDs
• Network access to OS Downloads API for downloads or remote listing
• Existing downloaded archives if you want to run offline without re-downloading
• Credentials in .env:
  • OS_PROJECT_API_KEY
  • OS_PROJECT_API_SECRET

If the required zip files already exist in your downloads directory, the build can now continue offline without contacting OS Data Hub. --list-only still requires network access because it queries remote package metadata.

Install from PyPI

pip install ukam-os-builder

Or with uv:

uv tool install ukam-os-builder

Run without installing (uvx)

You can run commands directly from PyPI without a permanent install:

uvx --from ukam-os-builder ukam-os-setup --help
uvx --from ukam-os-builder ukam-os-build --help

Example full run:

uvx --from ukam-os-builder ukam-os-setup --config-out config.yaml
uvx --from ukam-os-builder ukam-os-build --config config.yaml

After installation, CLI commands are available directly:

ukam-os-setup --help
ukam-os-build --help

Quick start

Workflow 1: CLI

1. Generate config with the setup wizard

ukam-os-setup --config-out config.yaml

This writes config.yaml and, by default, .env placeholders if .env does not already exist. The setup flow asks which source to use (ngd or abp) and stores it in config.yaml.

2. Add real credentials

Edit .env:

OS_PROJECT_API_KEY=your_api_key_here
OS_PROJECT_API_SECRET=your_api_secret_here

3. Run the full pipeline

ukam-os-build --config config.yaml

--config is the standard argument for selecting your configuration file.

Workflow 2: Python functions

from ukam_os_builder import create_config_and_env, run_from_config

create_config_and_env(
  config_out="config.yaml",
  env_out=".env",
  source="ngd",
  package_id="16331",
  version_id="104444",
  ngd_excluded_stems=["historicaddress"],
  abp_excluded_logical_statuses=[8],
)

run_from_config(config_path="config.yaml", step="all")

Inspect output variants

Use the reusable inspection function to find high-variant UPRNs in output parquet files:

from ukam_os_builder import inspect_flatfile_variants

result = inspect_flatfile_variants(config_path="config.yaml", top_offset=0, show=True)
print(result["selected_uprn"], result["variant_count"])

You can also import directly from the inspection module:

from ukam_os_builder.os_builder.inspect_results import inspect_flatfile_variants

result = inspect_flatfile_variants(config_path="config.yaml", top_offset=0, show=True)

CLI commands and key options

Command notes

• step only supports download and all to simplify usage. Use --overwrite to re-run a step with the same parameters.
• CLI overrides take precedence over values in config.yaml.
• By default, ukam-os-build loads .env from the same directory as your config, unless --env-file is supplied.

Full-run examples

Example A: guided setup then full run

ukam-os-setup --config-out config.yaml
ukam-os-build --config config.yaml

Example B: non-interactive setup and tuned full run

ukam-os-setup --source abp --config-out config.yaml --non-interactive --package-id <package_id> --version-id <version_id>
ukam-os-build --config config.yaml

Pipeline stages

1. download - fetch package metadata and zip files from OS Data Hub.
2. extract - convert NGD CSV members directly from downloaded zip files to parquet; ABP archives are extracted to CSV for the split step.
3. split - ABP only: split raw records and write only parquet staging files used by flatfile generation (street_descriptor, blpu, lpi, delivery_point, organisation, classification).
4. flatfile - transform and deduplicate into final output parquet file(s).

All stages are idempotent. Use --overwrite to regenerate outputs (--force is accepted as a backward-compatible alias).

ZIP extraction behavior

For NGD, the extract stage registers each downloaded ZIP with fsspec and DuckDB reads eligible CSV members from the archive while writing Parquet outputs. The decompressed CSV members are not persisted in paths.extracted_dir. Parquet files are written through temporary files and renamed into place after a successful conversion.

If the ZIP filesystem cannot be opened, registered, or read, the stage falls back to the legacy CSV extraction path. CSV parsing, type inference, conversion, and output-write errors are reported directly and do not trigger that fallback. Existing downloaded ZIPs can therefore be processed offline with:

ukam-os-build --config config.yaml --step extract

ABP continues to materialise CSV files because its split stage consumes raw CSV records.

Output

Final outputs are parquet files in paths.output_dir:

• Single chunk: ngd_for_uk_address_matcher.chunk_001_of_001.parquet
• Multi-chunk: ngd_for_uk_address_matcher.chunk_001_of_00N.parquet, ...chunk_00N_of_00N.parquet

Chunking reduces memory use by processing UPRNs in batches. The default is num_chunks: 10, and the union of all chunk files equals the single-chunk output. Increase num_chunks for very large canonical files or machines with limited RAM; set it to 1 only when a single output file is explicitly required.

Schemas

Data Sources

The pipeline processes these NGD address feature types:

• Built Address (add_gb_builtaddress) - Current physical addresses
• Pre-Build Address (add_gb_prebuildaddress) - Planned or future addresses
• Historic Address (add_gb_historicaddress) - Historical addresses
• Non-Addressable Object (add_gb_nonaddressableobject) - Excluded from output
• Royal Mail Address (add_gb_royalmailaddress) - PAF delivery points
• Alternate addresses (*_altadd) - Alternative address variants

Welsh language variants are extracted where available and appear as separate rows in the output.

By default, NGD Historic Address and ABP Historic LPI records are excluded. Ordnance Survey experts advised that historic addresses can make address matching worse, because many records are not comprehensive old-business history; they can include planning, placeholder, or later replaced address variants that incorrectly match current addresses. Labelled data checks also showed historic records causing incorrect matches. Historic records may still improve some use cases, but they should be used carefully because adding them can fix some matches while degrading others. For use cases that need older business or address history, old cuts of NGD BuiltAddress or AddressBase may be more appropriate.

To change NGD feature exclusions, set processing.ngd_excluded_stems in config.yaml or pass --ngd-excluded-stems. Valid values are builtaddress, prebuildaddress, historicaddress, nonaddressableobject, and royalmailaddress. When a feature stem is excluded, its matching alternate-address file is excluded too; for example, builtaddress excludes both add_gb_builtaddress and add_gb_builtaddress_altadd. To include NGD historic addresses, set ngd_excluded_stems: [].

To change ABP status exclusions, set processing.abp_excluded_logical_statuses or pass --abp-excluded-logical-statuses. Valid values are 1 (approved), 3 (alternative), 6 (provisional), and 8 (historic). To include ABP historical records, set abp_excluded_logical_statuses: [].

Deduplication

When the same UPRN and address combination appears in multiple sources, records are deduplicated using these internal priority rules:

Feature type priority:

1. Built Address (highest)
2. Pre-Build Address
3. Royal Mail Address
4. Historic Address
5. Non-Addressable Object (excluded)

Address status priority:

1. Approved (highest)
2. Provisional
3. Alternative
4. Historical

Build status priority:

1. Built Complete (highest)
2. Under Construction
3. Prebuild
4. Historic
5. Demolished

OS Downloads API

To use the OS Downloads API:

1. Set up an API key
2. Add your key to .env: OS_PROJECT_API_KEY=your_key_here
3. Find your datapackage ID and version ID from the OS Data Hub
4. Update config.yaml with the package and version IDs

API reference

Base URL: https://api.os.uk/downloads/v1
Authentication: Header - key: OS_PROJECT_API_KEY

1. List versions for a datapackage:
   GET /dataPackages/{package_id}/versions
   Pick the version ID from the response (field: id)

2. List files available for download:
   GET /dataPackages/{package_id}/versions/{version_id}
   Read downloads[] for fileName, size, md5, url

3. Download data:
   Use the url from downloads[] with ?key=YOUR_API_KEY appended

Config shape (config.yaml)

source:
  type: ngd  # or abp

paths:
  work_dir: ./data

os_downloads:
  package_id: "<your_package_id>"
  version_id: "<your_version_id>"
  connect_timeout_seconds: 30
  read_timeout_seconds: 300

processing:
  parquet_compression: zstd
  parquet_compression_level: 9
  num_chunks: 10
  sort_output_by_postcode: false
  ngd_excluded_stems:
    - historicaddress
  abp_excluded_logical_statuses:
    - 8
  # duckdb_memory_limit: "8GB"

parquet_compression and parquet_compression_level apply to intermediate and final Parquet files. Set sort_output_by_postcode: true to sort each final NGD chunk by postcode and unique_id; this can reduce file size but requires additional sorting work.

By default, the tool creates these directories under paths.work_dir:

• downloads: <work_dir>/downloads
• extracted: <work_dir>/extracted
• parquet: <work_dir>/parquet
• output: <work_dir>/output

paths:
  work_dir: ./data
  overrides:
    downloads_dir: ./somewhere/downloads
    extracted_dir: /mnt/fast/extracted
    parquet_dir: ./data/parquet
    output_dir: ./output

Smoke test

pytest tests/test_smoke.py

Related projects

• uk_address_matcher
• prepare_addressbase_for_address_matching
• OS Data Hub - package/version management and downloads