lcfm-shapefiles

Python wrapper around shapefiles used for the LCFM project.

Install

From PyPI:

uv add lcfm-shapefiles
# or: pip install lcfm-shapefiles

From VITO's internal Artifactory (VITO-only):

uv add --index https://artifactory.vgt.vito.be/artifactory/api/pypi/python-packages/simple lcfm-shapefiles

Use

import lcfm_shapefiles import list_filepaths, get_filepath
import geopandas as gpd

# List out all available files; you can index this list to get a path
lcfm_shapefiles.list_filepaths()

# Get path a specific file
fp = lcfm_shapefiles.get_filepath("LCFM_1p_S2-tiles.fgb")

# Import fgb into a geopandas data frame
gdf = gpd.read_file(fp)

Develop

# Build venv
uv sync

# Run tests
uv run pytest

# Format the code
uv run ruff format .

License

The Python source code is licensed under the MIT License - see LICENSE.

The shapefile/grid data shipped in src/lcfm_shapefiles/data/ is licensed under CC BY-SA 4.0, except two files derived from OpenStreetMap land polygons (generated by scripts/land_buffer.py), which are ODbL 1.0 licensed - see LICENSE-DATA for the full breakdown and attribution.

CI/CD

A Jenkins job is configured, that:

• Builds the package
• Runs the tests
• Uploads a 'dev build' package to Artifactory, e.g. 0.2.3.dev20250305+6
• Creates a promotion job.

When the build is successful, you should receive an email with a link to the promotion job. If you don't get this email (e.g., when Jenkins can't infer your email from your commits), you can find the link to the promotion job in the build logs, or here. When you build this promotion job ("Build with Parameters" and then "Build"), it:

• Tags the commit based on the package version
• Uploads the 'tag build' package to Artifactory, e.g. 0.2.3.

This CI/CD uses the general Python pipeline and was created through the Jenkins job builder. Check with the Software Operations team for more details on its configuration or help with build issues.

Steps to to deploy a new or update existing shapefiles

• Add or update shapefiles in src/lcfm_shapefiles/data.
• Update tests accordingly.
• Bump the package version in pyproject.toml.
• Commit and push the changes to master.
• If the build passes, trigger the promotion job.

A new version of the lcfm-shapefiles package should now be on Artifactory.

Use Git LFS storage

We are using the Github Large File Storage about to store fgb files This works alongside with the principle that big files should not be stored in the repository, and a pointer is stored in their place. The actual files are stored somewhere from LFS and are synchronized/cloned from our master branch in the vm at: /vitodata/vegteam/auxdata/LCFM/shapefiles-lfs

sudo dnf install -y git-lfs
git lfs install

If you need to pull the actual fgb files try:

git lfs pull

MD5 and Shapefile limits

To ensure validity of downloaded assets, multi-part upload limit is set to 5GB aligning with the Cloudferro's suggested max size recommended limit

Auto-Syncing Shapefiles to Terrascope

When triggered, the script /vitodata/vegteam/auxdata/LCFM/scripts/sync-shapefiles-terrascope.sh performs a syncing from the master branch As Terrascope network is not exposed publicly and github event-hooks can not be used, the trigger is a weekly defined cronjob:

Cronjob

Cronjob is added with:

crontab -e

and then usin VIM editor add:

0 0 * * 0 /vitodata/vegteam/auxdata/LCFM/scripts/sync-shapefiles-terrascope.sh >> /vitodata/vegteam/auxdata/LCFM/logs/cron.log 2>&1

confirm that cronjob is added

crontab -l