xcheck 0.1.2

Python package for 'XCheck: A Consistency-based validation framework of englacial layers annotations' across consecutive and intersecting radargram pairs using dot-product similarity.

XCheck: A Consistency-Based Validation Framework for Englacial Layer Annotations

This python package repository provides the implementation for "XCheck: A Consistency-Based Validation Framework for Englacial Layer Annotations" (Paper), accepted for publication at IEEE ICDMW 2025 proceedings. We introduce a dot-product-based layer consistency framework that validates annotated englacial layers between two radargrams observing the same ice volume — either two consecutive flight segments (overlap detected via GPS time) or two intersecting flight paths (crossing point supplied via a pre-computed CSV).

---

Introduction

Ice-penetrating radar surveys produce radargrams — 2-D depth profiles of englacial structure. Human or automated annotators trace internal reflection horizons (layers) that represent isochrones of ice deposition. When two flight lines observe the same ice at a crossing or overlap point, consistent annotations should agree in depth. XCheck operationalizes this constraint: it extracts binary depth masks from annotated layers, aligns them geometrically at the shared observation point, and uses a sliding dot-product test to find matching layer pairs. A high match count indicates annotation consistency; a low count flags potential mislabelling or pick drift.

XCheck handles two radargram relationship types:

• Consecutive — two sequential segments of the same flight line share a tail/head overlap detectable via GPS timestamp matching.
• Non-consecutive (intersecting) — two flight lines from different dates or paths cross at a known geographic point supplied by a pre-computed intersections CSV.

---

Methodology

Algorithm Steps

Requires: Binary Masks M1, M2 · GPS time sequences T1, T2 · Flight paths P1, P2 · Surface elevations S1, S2 · Depth axis D · Window size w

Step	Name	Description
1	Temporal & Spatial Relationship Detection	For consecutive pairs: find overlapping GPS time intervals between T1 and T2 and derive the crossing point from the midpoint of the overlap region using flight paths P1, P2. For intersecting pairs: read the crossing point directly from the intersections CSV.
2	Surface Normalization	Align each radargram column vertically using surface elevation S1/S2 relative to the depth axis D, correcting for topographic variation before comparison.
3	Midpoint Column Extraction	Extract a window of w columns around the crossing point from each binary mask, centered on the closest trace to the crossing coordinate.
4	Dot Product Matching	For each candidate layer row pair (i, j) within max_distance pixels, compute the dot product of their column windows. A pair is matched if the score meets dp_threshold.
5	Aggregate Matches	Collect all one-to-one matched layer pairs and report total match count and alignment percentage.

---

Installation

pip install xcheck

NDH_PythonTools (required for data loading)

XCheck uses NDH_PythonTools for .mat file loading and radar data processing. It must be installed manually since it is not a standard pip package:

git clone https://github.com/nholschuh/NDH_PythonTools.git

Add its parent directory to PYTHONPATH before running:

# Linux / macOS
export PYTHONPATH="/path/to/parent/of/NDH_PythonTools"

# Windows PowerShell
$env:PYTHONPATH = "C:\path\to\parent\of\NDH_PythonTools"

The pure matching algorithm (match_layers_with_dot_product) does not require NDH_PythonTools and works standalone on any NumPy arrays.

---

Directory Structure

pypi package/
├── src/xcheck/
│   ├── xcheck.py          # Core algorithms: loading, masking, GPS overlap,
│   │                      #   column extraction, layer matching, plotting, CLI
│   └── __init__.py        # Public API exports
├── pyproject.toml         # Package metadata and dependencies
├── README.md
└── dist/                  # Built wheel and source distribution

---

Quickstart

1. Command Line Interface

Consecutive radargrams (crossing derived automatically from GPS overlap):

xcheck --r1 Data_20120429_01_026.mat --r2 Data_20120429_01_027.mat \
       --layer_dir ./Nick-layer-data-mat/ \
       --radar_dir ./Nick-raw-radargram-mat/

Intersecting radargrams (crossing looked up from CSV):

xcheck --r1 Data_20120330_01_004.mat --r2 Data_20120511_01_054.mat \
       --intersections_csv ./Intersections_2012.csv \
       --layer_dir ./Nick-layer-data-mat/ \
       --radar_dir ./Nick-raw-radargram-mat/

Batch mode (all pairs in the intersections CSV):

xcheck --intersections_csv ./Intersections_2012.csv \
       --layer_dir ./Nick-layer-data-mat/ \
       --radar_dir ./Nick-raw-radargram-mat/

2. Python API

Layer matching only (no NDH_PythonTools needed):

import numpy as np
from xcheck import match_layers_with_dot_product

m1 = np.zeros((6, 5)); m1[[1, 3, 5], :] = 1
m2 = np.zeros((6, 5)); m2[[1, 3, 5], :] = 1

matches = match_layers_with_dot_product(m1, m2, max_distance=1,
                                        window_size=3, dp_threshold=1)
print(matches)  # [(1, 1), (3, 3), (5, 5)]

Full pipeline (requires NDH_PythonTools + data):

from xcheck import (
    load_and_process_radargram,
    load_log_gps_time,
    find_overlapping_intervals,
    extract_and_adjust_mask_columns_for_location,
    match_layers_with_dot_product,
)

r1, m1, d1 = load_and_process_radargram(
    "Data_20120429_01_026.mat",
    layer_dir="./Nick-layer-data-mat/",
    radar_dir="./Nick-raw-radargram-mat/")
r2, m2, d2 = load_and_process_radargram(
    "Data_20120429_01_027.mat",
    layer_dir="./Nick-layer-data-mat/",
    radar_dir="./Nick-raw-radargram-mat/")

log_t1 = load_log_gps_time("./Nick-raw-radargram-mat/Data_20120429_01_026.mat")
log_t2 = load_log_gps_time("./Nick-raw-radargram-mat/Data_20120429_01_027.mat")
overlaps1, overlaps2 = find_overlapping_intervals(log_t1, log_t2)

for iv1, iv2 in zip(overlaps1, overlaps2):
    mid1 = (iv1[0] + iv1[1]) // 2
    mid2 = (iv2[0] + iv2[1]) // 2
    mc1 = extract_and_adjust_mask_columns_for_location(
        r1['Longitude'][mid1], r1['Latitude'][mid1], r1, m1, d1, 20)
    mc2 = extract_and_adjust_mask_columns_for_location(
        r2['Longitude'][mid2], r2['Latitude'][mid2], r2, m2, d2, 20)
    matches = match_layers_with_dot_product(mc1, mc2)
    print(f"Matched layers: {matches}")

---

Parameters

Argument	Default	Description
--r1, --r2	—	Single-pair mode: filenames of the two radargrams
--lat, --lon	—	Explicit crossing coordinates for single-pair mode
--csv_path	—	Batch CSV with columns Radargram 1, Radargram 2, Latitude, Longitude
--intersections_csv	—	Pre-computed intersections CSV (used alone or to look up crossing coordinates)
--layer_dir	required	Directory of layer .mat files
--radar_dir	required	Directory of raw radargram .mat files
--layer_prefix	Layer_	Filename prefix prepended to layer files
--cols_side	20 / 3	Columns extracted each side of crossing (consecutive / non-consecutive)
--max_distance	4 / 3	Max depth-pixel gap between candidate layer rows
--window_size	7 / 3	Dot product window width (must be odd)
--dp_threshold	4 / 1	Minimum dot product score to confirm a match
--plot_overlaps	off	Plot GPS overlap regions on a map

Defaults shown as consecutive / non-consecutive — selected automatically based on whether a crossing point is available.

---

Default Thresholds by Radargram Type

Parameter	Consecutive	Non-Consecutive
cols_side	20 (41 columns)	3 (7 columns)
max_distance	4 px	3 px
window_size	7	3
dp_threshold	4	1

---

Methodological Note

GPS overlap detection compares log(GPS_time) between two radargrams with an absolute tolerance of 1e-8. Because d(log t) ≈ dt/t and GPS times are ~1×10⁹ seconds, this tolerance corresponds to matching raw timestamps within roughly 10 seconds. Consecutive flight segments share a small tail/head of overlapping traces where the crossing point is derived. Non-consecutive intersecting radargrams do not share GPS timestamps — their crossing point must be supplied via --intersections_csv or --lat/--lon.

---

Citation

Cite our work.

@INPROCEEDINGS{11415920,
  author={Hassan, Muhammad Behroze and Tama, Bayu Adhi and Purushotham, Sanjay and Janeja, Vandana P.},
  booktitle={2025 IEEE International Conference on Data Mining Workshops (ICDMW)}, 
  title={XCheck: A Consistency-Based Validation Framework for Englacial Layers Annotations}, 
  year={2025},
  volume={},
  number={},
  pages={76-85},
  keywords={Surveys;Radar remote sensing;Annotations;Ice sheets;Radar;Benchmark testing;Radar tracking;Robustness;Data mining;Remote sensing;Ice sheets;englacial layers;radargrams spatiotemporal relationship;validation metric;AI-ready dataset},
  doi={10.1109/ICDMW69685.2025.00015}
}

---

References

M. B. Hassan, B. A. Tama, S. Purushotham and V. P. Janeja, "XCheck: A Consistency-Based Validation Framework for Englacial Layers Annotations," 2025 IEEE International Conference on Data Mining Workshops (ICDMW), Washington, DC, USA, 2025, pp. 76-85, doi: 10.1109/ICDMW69685.2025.00015.

---

License

MIT