This project has been archived by its maintainers, and is no longer receiving any updates.
geodata-sync
Open GIS data loaded into PostGIS, and kept up to date, from YAML files. Say which datasets you want, where they come from and how often to check them; geodata sync downloads each new version, loads it into staging tables, checks it, backs up the old tables and swaps the new ones in, in one transaction.
- Find data: the Ordnance Survey's OpenData downloads, CKAN portals (data.gov.uk and many city, regional and national ones), WFS and OGC API Features services, or plain links.
geodata discoverlists what a source offers and the layers inside a download. - Load it safely: staging, invalid geometries left out, row counts checked, the old tables kept as backups; a load that fails changes nothing.
- Keep it up to date: each dataset has a schedule (
daily,weekly,monthlyor a cron expression);geodata syncrun nightly checks those that are due, and loads only a new version. - Rasters too: a raster download (OS Terrain 50's 2,800 grid tiles, say) becomes one Cloud-Optimised GeoTIFF.
It only loads data. To publish the tables in GeoServer or GeoServer Cloud, use GeoServer ConfigOps: configops refresh brings GeoServer up to date after a load.
Status: early. Expect breaking changes.
Install
pip install geodata-sync # the geodata command; it needs GDAL's tools (apt install gdal-bin)
docker run --rm ghcr.io/yibbzz/geodata-sync --help # or its image, GDAL included
Quick start
With docker and geodata. This starts a PostGIS database (examples/compose.yaml) and loads examples/quickstart: Surrey Heath's conservation areas, from data.gov.uk.
apiVersion: geodata/v1alpha1
kind: Project
metadata: { name: quickstart }
spec:
database:
connection: { secret: database } # keys: host, port, database, username, password
crs: EPSG:27700
---
apiVersion: geodata/v1alpha1
kind: Source
metadata: { name: data-gov-uk }
spec: { type: ckan, url: https://data.gov.uk }
---
apiVersion: geodata/v1alpha1
kind: Dataset
metadata: { name: conservation-areas }
spec:
source: data-gov-uk
product: conservation-areas125 # `geodata discover data-gov-uk --search "conservation areas"` finds it
update: monthly
tables:
conservation_areas: {} # the layer in the download; table conservation_areas_conservation_areas
docker compose -f examples/compose.yaml up -d --wait
cat > examples/quickstart/secrets.yaml <<'EOF' # git-ignored
database: { host: localhost, port: "5433", database: postgres, username: postgres, password: postgres }
EOF
geodata sync -C examples/quickstart
# conservation-areas: loaded 2026-11-11 in 0s: data_layers.conservation_areas_conservation_areas (10 rows)
# note: 1 invalid geometry left out
geodata status -C examples/quickstart
# DATASET UPDATE LOADED AT ROWS NEXT CHECK LAST CHECK
# conservation-areas monthly 2026-11-11 2026-11-13 08:51 10 2026-12-13 07:51 success
geodata sync -C examples/quickstart
# conservation-areas: not due (monthly)
docker compose -f examples/compose.yaml down -v removes it all. docs/spec.md describes every kind, field and command.
Examples
| Project | What it holds |
|---|---|
examples/quickstart |
The quick start above: one dataset from data.gov.uk, 10 polygons. Loads in seconds |
examples/uk-basemap |
OS Open Roads (7.3M features), OS Boundary-Line, the Environment Agency's flood zones (415k polygons) and the conservation areas. Downloads about 3 GB; the first load takes 20 minutes or so. GeoServer ConfigOps's examples/uk-basemap publishes them |
examples/terrain |
OS Terrain 50 (heights of Great Britain, 50 m) made into one Cloud-Optimised GeoTIFF |
How to
Find a dataset:
geodata discover os-downloads # OS products
geodata discover data-gov-uk --search "flood zones" -C my-data # a Source in the project
geodata discover data-gov-uk <product> -C my-data # its files and the layers inside
geodata validate my-data # problems come out as file:line
Load now rather than when due: geodata sync -C my-data <dataset> (--force loads it even when the version hasn't changed).
Refresh GeoServer after a load: geodata sync -C my-data --loaded loaded.txt writes what it loaded (schema.table, or a raster's path), and configops refresh -C my-geoserver --from loaded.txt works out the bounds of the layers on them again and clears their cached tiles.
Keep the logins out of the files. A project names Secrets (connection: { secret: database }); geodata reads each from a git-ignored secrets.yaml in the project folder, from GEODATA_<SECRET>_<KEY> environment variables, or from files $GEODATA_SECRETS_DIR/<secret>/<key> (how Kubernetes mounts Secrets).
Run it in Kubernetes: docs/kubernetes.md has a nightly CronJob that loads, then refreshes GeoServer.
docs/sources-notes.md has what we learned about the sources: big layers over WFS or OGC API Features don't load reliably, so use a file download when there is one.
Testing
python3 -m venv .venv && .venv/bin/pip install -e '.[dev]'
.venv/bin/pytest # the PostGIS tests skip without one
docker run -d --rm --name test-postgis -p 55432:5432 \
-e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=gis postgis/postgis:18-3.7
TEST_DB_HOST=localhost TEST_DB_PORT=55432 .venv/bin/pytest -m integration
CI runs the tests with a real PostGIS (test), shellcheck and a schema check of the Kubernetes example (lint), and the image build (image).
Releasing
Set the version in pyproject.toml and geodata_sync/__init__.py, merge, then push the tag v<version>: the Release workflow publishes the package to PyPI (geodata-sync), the image to ghcr.io/yibbzz/geodata-sync, and a GitHub release.
Licence and data attribution
MIT (see LICENSE). This code started in GeoServer ConfigOps, where it was the sync and discover commands.
Data you load keeps its own licence. OS OpenData is released under the Open Government Licence v3.0 and needs the attribution "Contains OS data © Crown copyright and database right <year>" wherever it's shown.
Metadata
Release files for geodata-sync 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| geodata_sync-0.1.0.tar.gz | 62.1 kB | With provenance | Details for geodata_sync-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| geodata_sync-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for geodata_sync-0.1.0-py3-none-any.whl |
Total release size: 115.4 kB
Release files / geodata_sync-0.1.0.tar.gz
| Download URL | geodata_sync-0.1.0.tar.gz |
|---|---|
| Size | 62.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1e7738630997fc186c9226582ff5b513f72e7938d9b53e525e84ba50a9fb0701
|
|
BLAKE2b-256 checksum How to use checksums |
2ecba36152329d2df5549abb2d4a69a08a36970f713d0a88ae8a9232fcdb493c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logRelease files / geodata_sync-0.1.0-py3-none-any.whl
| Download URL | geodata_sync-0.1.0-py3-none-any.whl |
|---|---|
| Size | 53.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2d9fe74c2c23b1efe294cca786458f5ad552a4bb2206c4400a6547c4a90b3150
|
|
BLAKE2b-256 checksum How to use checksums |
67e0b887be8a1c06ee2f9488037428995eced2fdeab6d5e029a7bc1593256ffb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logSecurity for geodata-sync 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for geodata_sync-0.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 13, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.