vector2dggs

Python-based CLI tool to index vector files to DGGS in parallel, writing out to Parquet.

This is the vector equivalent of raster2dggs.

Currently this tool supports the following DGGSs:

• H3
• rHEALPix
• S2
• A5

... and the following geocode systems:

• Geohash

Contributions (especially for other DGGSs), suggestions, bug reports and strongly worded letters are all welcome.

Installation

This tool makes use of optional extras to allow you to install a limited subset of DGGSs.

If you want all possible:

pip install vector2dggs[all]

If you want only a subset, use the pattern pip install vector2dggs[rhp] (for one) or pip install vector2dggs[h3,s2] (for multiple).

A bare pip install vector2dggs will not install any DGGS backends.

PostgreSQL/PostGIS input requires the postgres extra (e.g. pip install vector2dggs[h3,postgres]); it is included in all.

Usage

All commands (h3, rhp, s2, a5, geohash) share the same interface:

vector2dggs <dggs> [OPTIONS] VECTOR_INPUT OUTPUT_DIRECTORY

VECTOR_INPUT may be a local file (anything GDAL can read), a remote URI or GDAL virtual path (e.g. https://…, /vsizip/…), or a PostgreSQL/PostGIS connection URL (with -lyr naming the table). OUTPUT_DIRECTORY is written as an Apache Parquet data store: a directory with one file per partition.

• -r/--resolution: the target DGGS resolution. Each output row is one (feature, cell) pair; a cell is included when its centre falls inside the feature, uniformly across all backends.
• -pr/--parent_res: a coarser resolution used to partition the output (hive directories such as h3_03=…); defaults to a fixed offset below the target resolution.
• -id/--id_field: the feature identifier carried into the output; defaults to a synthetic index. Rows sharing an id are treated as one feature.
• -co/--compact (requires -id): merges complete sets of sibling cells belonging to one feature, to no coarser than the parent resolution. Compacted output expands back to exactly the full-resolution result.
• --geo: plain Parquet by default; point or polygon writes GeoParquet (v1.1.0) cell geometries instead.

The full reference (vector2dggs h3 --help; the other commands differ only in their resolution ranges):

Usage: vector2dggs h3 [OPTIONS] VECTOR_INPUT OUTPUT_DIRECTORY

  Ingest a vector dataset and index it to the H3 DGGS.

  VECTOR_INPUT is the path to input vector geospatial data. OUTPUT_DIRECTORY
  should be a directory, not a file or database table, as it will instead be the
  write location for an Apache Parquet data store.

Options:
  -v, --verbosity LVL             Either CRITICAL, ERROR, WARNING, INFO or DEBUG
                                  [default: INFO]
  -r, --resolution [0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15]
                                  H3 resolution to index  [required]
  -pr, --parent_res [0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15]
                                  H3 parent resolution for the output partition.
                                  Defaults to resolution - 6
  -id, --id_field TEXT            Field to use as an ID; defaults to a
                                  constructed single 0...n index on the original
                                  feature order.
  -k, --keep_attributes           Retain attributes in output. The default is to
                                  create an output that only includes H3 cell ID
                                  and the ID given by the -id field (or the
                                  default index ID).
  -crs, --cut_crs INTEGER         Set the coordinate reference system (CRS) used
                                  for cutting large geometries (see
                                  `--cut_threshold`). Defaults to the same CRS
                                  as the input. Should be a valid EPSG code.
  -c, --cut_threshold FLOAT       Cutting up large geometries into smaller
                                  geometries based on a target area. Units are
                                  assumed to match the input CRS units unless
                                  `--cut_crs` is also given, in which case units
                                  match the units of the supplied CRS. If left
                                  unspecified, the threshold defaults to the
                                  area of a few thousand cells of the target
                                  resolution (a benchmarked balance of
                                  parallelism against per-piece overhead),
                                  converted into the squared units of the
                                  cutting CRS. A threshold of 0 will skip
                                  bisection entirely (effectively ignoring
                                  --cut_crs).
  -t, --threads INTEGER RANGE     Amount of threads used for operation
                                  [default: (CPU count - 1); x>=1]
  -cp, --compression TEXT         Compression method to use for the output
                                  Parquet files. Options include 'snappy',
                                  'gzip', 'brotli', 'lz4', 'zstd', etc. Use
                                  'none' for no compression.  [default: snappy]
  -lyr, --layer TEXT              Name of the layer or table to read when using
                                  an input that supports layers or tables
  -g, --geom_col TEXT             Column name to use when using a spatial
                                  database connection as input  [default: geom]
  --geo [none|point|polygon]      Select geometry encoding for the output:
                                  'none' for regular Parquet (no GeoParquet
                                  metadata), or 'point'/'polygon' to write
                                  GeoParquet (v1.1.0) with the corresponding
                                  geometry type.  [default: none]
  --tempdir PATH                  Temporary data is created during the execution
                                  of this program. This parameter allows you to
                                  control where this data will be written.
                                  [default: (system temp dir)]
  -co, --compact                  Compact the H3 cells up to the parent
                                  resolution. Compaction requires an id_field.
  -o, --overwrite
  --version                       Show the version and exit.
  --help                          Show this message and exit.

vector2dggs is a command-line tool; the underlying Python API (vector2dggs.common.index) can be called directly but is not a stable, supported interface.

Visualising output

Output is in the Apache Parquet format, a directory with one file per partition. With --geo point or --geo polygon output will be written as GeoParquet (v1.1.0) with the respective geometry types. GeoParquet can be visualised using desktop GIS tools.

The Apache Parquet output is indexed by an ID column (which you can specify), so it should be ready for two intended use-cases:

• Joining attribute data from the original feature-level data onto computed DGGS cells.
• Joining other data to this output on the DGGS cell ID. (The output has a column like {dggs}_\d, e.g. h3_09 or h3_12 according to the target resolution, zero-padded to account for the maximum resolution of the DGGS).

Compaction

Compaction is supported with the -co/--compact argument. The result respects overlapping polygons by considering each feature independently. (In the below example output for rHEALPix, cells are shown with opacity; overlap is visible where there is a darker shade.) This does mean that the index of the result is not necessarily unique (unless your input is a vector coverage, i.e. it does not have overlaps.)

For development

In brief, to get started:

• Install Poetry
• Create and populate the virtual environment with poetry install. This will install necessary dependencies.

No system GDAL is required: vector data is read via pyogrio, whose wheels bundle GDAL. If you need a GDAL driver that pyogrio's bundled build lacks, build pyogrio from source against your own GDAL.

• Subsequently, activate the virtual environment with eval "$(poetry env activate)".

If you run poetry install -E all --with dev and activate the environment with eval "$(poetry env activate)", the CLI tool will be aliased so you can simply use vector2dggs rather than poetry run vector2dggs.

For partial backend support you can consider poetry install --with dev -E h3 -E s2 etc. To check what is installed: poetry show --tree.

Alternatively, it is also possible to install using pip with pip install -e ., and bypass Poetry.

Code formatting, linting and type checking

Please run black ., ruff check . and mypy vector2dggs/ before committing. All three are enforced in CI.

Tests

Tests are included. Some tests (covering the PostgreSQL/PostGIS input path) spin up a throwaway PostGIS container via Docker; they'll be skipped automatically if Docker isn't available, rather than failing the run.

To run them, activate the Poetry environment first (eval "$(poetry env activate)"), then run:

python -m pytest -n auto tests/

Or without activating the shell:

poetry run pytest -n auto tests/

To test a specific DGGS:

python -m pytest tests/test_runthrough.py -k "a5" -v

Test data are included at tests/data/.

Example commands

With a local GPKG:

vector2dggs h3 -v DEBUG -id title_no -r 12 -o ~/Downloads/nz-property-titles.gpkg ~/Downloads/nz-property-titles.parquet

With a PostgreSQL/PostGIS connection:

vector2dggs h3 -v DEBUG -id ogc_fid -r 9 -pr 5 -t 4 --overwrite -lyr topo50_lake postgresql://user:password@host:port/db ./topo50_lake.parquet

Citation

Citation metadata is maintained in CITATION.cff. GitHub renders this as a "Cite this repository" button on the repository page (top-right of the About panel), which provides ready-to-copy BibTeX and APA formats.