FIT Change Detector

Compare two sets of geo-data and report on the differences.

Installation

Install with pip:

pip install fit_changedetector

To use the scripts on ArcGIS Pro systems, installation is not required but uv must be available at the Windows command prompt. See the arcgis/ folder, and its README, for the scripts and setup instructions.

Usage

Python module

The primary function of interest is gdf_diff():

import geopandas
import fit_changedetector as fcd

# read the data
df_a = geopandas.read_file(in_file_a, layer=layer_a)
df_b = geopandas.read_file(in_file_b, layer=layer_b)

# compare the two dataframes
diff = fcd.gdf_diff(
    df_a,
    df_b,
    <primary_key>,
    fields=<fields_to_compare>,
    precision=<precision>,
    suffix_a="a",
    suffix_b="b",
)

gdf_diff returns a dictionary having the keys noted below. Dictionary values are geopandas GeoDataFrames holding the corresponding records.

Dictionary keys:

Schemas for records contained in NEW, DELETED, UNCHANGED are as per the source data. DUPLICATES records retain their source schema too, plus a _fcd_source_ column (suffix_a/suffix_b) identifying which source each record was dropped from. Schemas for records contained in the MODIFIED keys include only columns where a change has occurred. For example, these are some "modified attributes" records, with "_a" suffix for values from the primary dataset, and "_b" suffix for values from the secondary dataset:

>>> diff["MODIFIED_ATTR"]
  id       park_name_a                park_name_b parkclasscode_a parkclasscode_b
0  3  Mars Street Park        Jupiter Street Park             NaN             NaN
1  6      Mayfair Blue              Mayfair Green              BL             GRN
2  7                    Quadra Heights Playground             NaN             NaN
3  9               NaN                        NaN              RP             PND

CLI

$ changedetector --help
Usage: changedetector [OPTIONS] COMMAND [ARGS]...

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  add-hash-key  Read input data, compute hash, write to new file
  diff          Compare two datasets, printing a JSON summary to stdout
  diff2gdb      Compare two datasets, writing results to .gdb

$ changedetector add-hash-key --help
Usage: changedetector add-hash-key [OPTIONS] IN_FILE OUT_FILE

  Read input data, compute hash, write to new file

Options:
  --in-layer TEXT           Name of layer to add hashed primary key
  -nln, --out-layer TEXT    Output layer name
  -hk, --hash-key TEXT      Name of new column containing hashed data
  -d, --drop-null-geometry  Drop records with null geometry. Only valid when the
                            geometry field is included in --hash-fields
  -hf, --hash-fields TEXT   Comma separated list of fields to hash. Include the
                            geometry field's name (typically 'geometry') to
                            include geometry in the hash  [required]
  -p, --precision FLOAT     Coordinate precision for geometry hash and
                            comparison. Default=0.01
  --crs TEXT                Coordinate reference system to use when hashing
                            geometries (eg EPSG:3005)
  -v, --verbose             Increase verbosity.
  -q, --quiet               Decrease verbosity.
  --help                    Show this message and exit.

$ changedetector diff --help
Usage: changedetector diff [OPTIONS] IN_FILE_A IN_FILE_B

  Compare two datasets, printing a JSON summary to stdout

  Same comparison as `diff2gdb`, but for when spatial output isn't needed -
  prints a JSON summary instead of writing a .gdb: record counts per
  NEW/DELETED/UNCHANGED/MODIFIED_* category, plus the primary key value(s)
  present in each category other than UNCHANGED (use --count to omit the key
  lists and print just the counts). Use --out-file to write the JSON to a file
  instead of stdout.

  To read GeoJSON from stdin, specify "-" for IN_FILE_A

Options:
  --layer-a TEXT             Name of layer to use within in_file_a (not valid if
                             reading from stdin/parquet)
  --layer-b TEXT             Name of layer to use within in_file_b (not valid if
                             reading from parquet)
  -f, --fields TEXT          Comma separated list of fields to compare (do not
                             include primary key)
  -if, --ignore-fields TEXT  Comma separated list of fields to ignore
  -pk, --primary-key TEXT    Name of primary key column, common to both datasets
                             - for a composite key, use --hash-fields instead to
                             generate one from multiple fields
  -hk, --hash-key TEXT       Name of new column to add as hash key
  -hf, --hash-fields TEXT    Comma separated list of fields to hash, when no
                             --primary-key is given - required in that case.
                             Include the geometry field's name (typically
                             'geometry') to include geometry in the hash
  -p, --precision FLOAT      Coordinate precision for geometry hash and
                             comparison. Default=0.01
  -a, --suffix-a TEXT        Suffix to append to column names from data source A
                             when comparing attributes
  -b, --suffix-b TEXT        Suffix to append to column names from data source B
                             when comparing attributes
  -d, --drop-null-geometry   Drop records with null geometry. Only valid when
                             the geometry field is included in --hash-fields -
                             has no effect (and is rejected) when --primary-key
                             is supplied, since no hash key is generated in that
                             case
  --crs TEXT                 Coordinate reference system to use when hashing
                             geometries (eg EPSG:3005)
  --allow-duplicates         Do not fail on a duplicated primary key - instead,
                             drop all but the first occurrence of each
                             duplicated key from the source it was found in, and
                             include the dropped records in a DUPLICATES
                             category/layer of the output. Not applied to a pure
                             geometry hash (no primary key, and --hash-fields
                             hashes on the geometry field alone) - a duplicate
                             there always fails, since geometry alone can't
                             reliably pair records between datasets when more
                             than one shares a location
  --no-promote-multi         Stricter geometry checking - compare geometries as-
                             is. By default, when the sources mix single and
                             multipart geometries of the same type (eg Point and
                             MultiPoint), all geometries are promoted to
                             multipart before comparing, so a feature stored
                             single-part in one source and multi-part in the
                             other is UNCHANGED - with this option it is
                             MODIFIED_GEOM
  --strict-types             Require compared field types to match exactly. By
                             default, integer fields of differing width (eg
                             Integer vs Integer64) are compared as the smallest
                             integer type holding both, with a warning
  -c, --count                Print only record counts, omitting the primary key
                             values in each category
  -o, --out-file PATH        Path to write JSON summary to, instead of printing
                             to stdout
  -v, --verbose              Increase verbosity.
  -q, --quiet                Decrease verbosity.
  --help                     Show this message and exit.

$ changedetector diff2gdb --help
Usage: changedetector diff2gdb [OPTIONS] IN_FILE_A IN_FILE_B

  Compare two datasets, writing results to .gdb

  To read GeoJSON from stdin, specify "-" for IN_FILE_A

Options:
  --layer-a TEXT             Name of layer to use within in_file_a (not valid if
                             reading from stdin/parquet)
  --layer-b TEXT             Name of layer to use within in_file_b (not valid if
                             reading from parquet)
  -f, --fields TEXT          Comma separated list of fields to compare (do not
                             include primary key)
  -if, --ignore-fields TEXT  Comma separated list of fields to ignore
  -pk, --primary-key TEXT    Name of primary key column, common to both datasets
                             - for a composite key, use --hash-fields instead to
                             generate one from multiple fields
  -hk, --hash-key TEXT       Name of new column to add as hash key
  -hf, --hash-fields TEXT    Comma separated list of fields to hash, when no
                             --primary-key is given - required in that case.
                             Include the geometry field's name (typically
                             'geometry') to include geometry in the hash
  -p, --precision FLOAT      Coordinate precision for geometry hash and
                             comparison. Default=0.01
  -a, --suffix-a TEXT        Suffix to append to column names from data source A
                             when comparing attributes
  -b, --suffix-b TEXT        Suffix to append to column names from data source B
                             when comparing attributes
  -d, --drop-null-geometry   Drop records with null geometry. Only valid when
                             the geometry field is included in --hash-fields -
                             has no effect (and is rejected) when --primary-key
                             is supplied, since no hash key is generated in that
                             case
  --crs TEXT                 Coordinate reference system to use when hashing
                             geometries (eg EPSG:3005)
  --allow-duplicates         Do not fail on a duplicated primary key - instead,
                             drop all but the first occurrence of each
                             duplicated key from the source it was found in, and
                             include the dropped records in a DUPLICATES
                             category/layer of the output. Not applied to a pure
                             geometry hash (no primary key, and --hash-fields
                             hashes on the geometry field alone) - a duplicate
                             there always fails, since geometry alone can't
                             reliably pair records between datasets when more
                             than one shares a location
  --no-promote-multi         Stricter geometry checking - compare geometries as-
                             is. By default, when the sources mix single and
                             multipart geometries of the same type (eg Point and
                             MultiPoint), all geometries are promoted to
                             multipart before comparing, so a feature stored
                             single-part in one source and multi-part in the
                             other is UNCHANGED - with this option it is
                             MODIFIED_GEOM
  --strict-types             Require compared field types to match exactly. By
                             default, integer fields of differing width (eg
                             Integer vs Integer64) are compared as the smallest
                             integer type holding both, with a warning
  -o, --out-file PATH        Path to output file, defaults to
                             ./changedetector_YYYYMMDD_HHMM.gdb
  -i, --dump-inputs          Dump input layers (with new hash key) to output
                             .gdb
  -v, --verbose              Increase verbosity.
  -q, --quiet                Decrease verbosity.
  --help                     Show this message and exit.

.gdb output (diff2gdb, add-hash-key) is written with GDAL's TARGET_ARCGIS_VERSION=ARCGIS_PRO_3_2_OR_LATER option, so 64-bit integer fields are written as 64-bit integers (rather than as floating point, which cannot exactly represent every value above 2^53 - eg large ids). Reading these fields requires ArcGIS Pro 3.2 or later.

Examples

Compare the test datasets using their known primary key:

$ changedetector diff2gdb -v \
    tests/data/parks_a.geojson \
    tests/data/parks_b.geojson \
    -pk id

Compare the test datasets, using a hash of geometry and the column park_name as synthetic primary key, written to new_hash_column (include the geometry field's name, typically geometry, in --hash-fields to fold geometry into the hash - omit it to hash on attributes only):

$ changedetector diff2gdb -v \
    tests/data/parks_a.geojson \
    tests/data/parks_b.geojson \
    -hf park_name,geometry \
    -hk new_hash_column

IN_FILE_A may be - to read GeoJSON from stdin instead of a file, e.g. to compare a database export against a file on disk without writing the export to disk first:

$ ogr2ogr -f GeoJSON /vsistdout/ PG:"dbname=mydb" -sql "SELECT * FROM my_table" | \
    changedetector diff2gdb -v - tests/data/parks_b.geojson -pk id

diff prints record counts per category plus the primary key value(s) present in each (except UNCHANGED) by default; add --count/-c to print just the counts:

$ changedetector diff tests/data/parks_a.geojson tests/data/parks_b.geojson -pk id
{"NEW": 1, "DELETED": 1, "UNCHANGED": 1, "MODIFIED_BOTH": 1, "MODIFIED_ATTR": 4, "MODIFIED_GEOM": 1, "keys": {"NEW": ["8"], "DELETED": ["2"], "MODIFIED_BOTH": ["5"], "MODIFIED_ATTR": ["3", "6", "7", "9"], "MODIFIED_GEOM": ["4"]}}

$ changedetector diff tests/data/parks_a.geojson tests/data/parks_b.geojson -pk id --count
{"NEW": 1, "DELETED": 1, "UNCHANGED": 1, "MODIFIED_BOTH": 1, "MODIFIED_ATTR": 4, "MODIFIED_GEOM": 1}

Usage notes

Layer options are not valid for stdin and parquet sources - streams and parquet files have no concept of multiple layers.

A partitioned parquet dataset (a directory of many .parquet files) is not supported as a single source - diff/diff2gdb load an entire source into memory regardless of format, so there's no benefit to teaching them to read a partition directory as one dataset. Instead, run the command once per file, e.g. for two partitioned datasets with matching partition filenames:

$ for part in dataset_a/*.parquet; do
    name=$(basename "$part" .parquet)
    changedetector diff2gdb -v \
        "$part" \
        "dataset_b/${name}.parquet" \
        -pk id \
        -o "output_${name}.gdb"
  done

Neither diff nor diff2gdb have a bounding box / spatial filtering option, and won't - applying a bbox filter independently to each source risks reporting spurious NEW/DELETED records for anything that moved across the boundary between the two snapshots being compared, rather than genuinely appearing/disappearing from the source. Filter with another tool first, e.g.:

$ ogr2ogr -f GeoJSON /vsistdout/ dataset_a.gpkg -spat 1150000 470000 1200000 500000 | \
    changedetector diff2gdb -v - dataset_b.gpkg -pk id

Subtleties to geometry change detection

Prior to comparing geometries, the tool will:

• normalize geometries (vertex order/starting point or ring winding direction)
• promote single/multipart types to multipart, when the sources (within or between them) contain both variants of a type - so a feature stored single-part in one source and multi-part in the other is UNCHANGED. Use --no-promote-multi for stricter checking, comparing geometries as-is (such a feature is then MODIFIED_GEOM)
• apply coordinate precision tolerance (-p/--precision, default 0.01)

On the other hand, a new vertex in an otherwise unchanged geometry will be considered as MODIFIED_GEOM. See geopandas geom_equals_exact for details.

Curved geometry types (CIRCULARSTRING, COMPOUNDCURVE, CURVEPOLYGON, etc, as found in some .gdb sources) are not supported - only POINT, LINESTRING, POLYGON and their MULTI* equivalents are (#66). GDAL segments curves into their linear approximation on read, so a curved source may appear to work, but the precision of that approximation is not controlled by diff/diff2gdb and results involving curved input should not be relied on.

Development and testing

Uses uv for dependency management:

$ git clone https://github.com/bcgov/FIT_changedetector.git
$ cd FIT_changedetector
$ uv sync --extra test
$ uv run pytest

Note that if testing only the ArcGIS scripts, uv sync is not necessary:

• clone the repo to get the latest script tool python files (in /arcgis)
• place all the .py files together in the desired folder
• update corresponding ArcGIS toolbox(es) as needed
• create a spec_override.txt file next to changedetector_common.py containing the git ref of this module you'd like to test against - gitignored, so it survives a git pull untouched (see the arcgis readme for an example)

Releases

The package version is derived from git tags via setuptools_scm - there's no hardcoded version to bump in source. Between tags, the version is a dev version derived from the most recent tag plus commit count/hash (e.g. 0.1.0a2.dev5+g1234abc).

To cut a release, push a tag matching vX.Y.Z (e.g. v0.2.0). This triggers .github/workflows/release.yml, which builds and publishes the package to PyPI, creates a GitHub Release, builds/attaches a version-pinned copy of the ArcGIS tools, and bumps arcgis/changedetector_common.py's FIT_CHANGEDETECTOR_SPEC to match on main - see arcgis/README.md.