fiboa-cli 0.21.0

CLI tools such as validation and file format conversion for fiboa.

fiboa CLI

A command-line interface (CLI) for working with fiboa files.

• Getting Started
• Documentation / Commands
• Development

Getting Started

In order to make working with fiboa easier we have developed command-line interface (CLI) tools such as inspection, validation and file format conversions.

Installation

Using Pixi (Recommended)

This project uses Pixi for dependency management. Install Pixi first, then:

# Clone the repository and navigate to it
git clone https://github.com/fiboa/cli.git
cd cli

# Install all dependencies
pixi install

# Run the CLI
pixi run fiboa

Using pip

Alternatively, you can install from PyPI with Python 3.11 or any later version:

pip install fiboa-cli

Execute a command

After the installation you should be able to run the following command: fiboa (or pixi run fiboa if using Pixi)

You should see usage instructions and available commands for the CLI.

fiboa CLI supports various commands to work with the files:

• fiboa CLI
  • Getting Started
    • Installation
      • Using Pixi (Recommended)
      • Using pip
    • Execute a command
  • Commands
    • Validation
    • Create fiboa GeoParquet from GeoJSON
    • Create fiboa GeoJSON from GeoParquet
    • Inspect fiboa GeoParquet file
    • Merge fiboa GeoParquet files
    • Create JSON Schema from fiboa Schema
    • Validate a Vecorel Schema
    • Improve a fiboa Parquet file
    • Update an extension template with new names
    • Converter for existing datasets
    • Publish datasets to source coop or your own s3 repository
  • Development
    • Implement a converter
  • Run in Docker

Commands

Validation

To validate a fiboa GeoParquet or GeoJSON file, you can for example run:

• GeoJSON: fiboa validate example.json --collection collection.json
• GeoParquet: fiboa validate example.parquet --data

Check fiboa validate --help for more details.

The validator also supports remote files.

• http:// or https://: no further configuration is needed.
• s3://: With Pixi, run pixi install -e s3 or with pip, run pip install fiboa-cli[s3] and you may need to set environment variables. Refer to the s3fs credentials documentation for how to define credentials.
• gs://: With Pixi, run pixi install -e gcs or with pip, run pip install fiboa-cli[gcs]. By default, gcsfs will attempt to use your default gcloud credentials or, attempt to get credentials from the google metadata service, or fall back to anonymous access.

Create fiboa GeoParquet from GeoJSON

To create a fiboa-compliant GeoParquet for a fiboa-compliant set of GeoJSON files containing Features or FeatureCollections, you can for example run:

• fiboa create-geoparquet geojson/example.json -o example.parquet -c geojson/collection.json

Check fiboa create-geoparquet --help for more details.

Create fiboa GeoJSON from GeoParquet

To create one or multiple fiboa-compliant GeoJSON file(s) for a fiboa-compliant GeoParquet file, you can for example run:

• GeoJSON FeatureCollection: fiboa create-geojson example.parquet -o dest-folder

• GeoJSON Features (with indentation and max. 100 features): fiboa create-geojson example.parquet -o dest-folder -n 100 -i 2 -f

  fiboa create-geojson example.parquet -o dest-folder -n 100 -i 2 -f

Check fiboa create-geojson --help for more details.

Inspect fiboa GeoParquet file

To look into a fiboa GeoParquet file to get a rough understanding of the content, the following can be executed:

• fiboa describe example.parquet

Check fiboa describe --help for more details.

Merge fiboa GeoParquet files

Merges multiple fiboa datasets to a combined fiboa dataset:

• fiboa merge ec_ee.parquet ec_lv.parquet -o merged.parquet -e https://fiboa.org/hcat-extension/v0.1.0/schema.yaml -i ec:hcat_name -i ec:hcat_code -i ec:translated_name

Check fiboa merge --help for more details.

Create JSON Schema from fiboa Schema

To create a JSON Schema for a fiboa Schema YAML file, you can for example run:

• fiboa jsonschema example.json --id=https://vecorel.org/specification/v0.3.0/geojson/schema.json -o schema.json

Check fiboa jsonschema --help for more details.

Validate a Vecorel Schema

To validate a Vecorel Schema YAML file, you can for example run:

• fiboa validate-schema schema/schema.yaml

Check fiboa validate-schema --help for more details.

Improve a fiboa Parquet file

Various "improvements" can be applied to a fiboa GeoParquet file. The commands allows to

• detect fiboa-0.2 files and convert them to fiboa-0.3
• change the CRS (--crs)
• change the GeoParquet version (-gp1) and compression (-pc)
• add/fill missing perimeter/area values (-sz)
• fix invalid geometries (-g)
• rename columns (-r)
• add HCAT columns (--hcat=mapping.csv) based on the input crop:code (from crop-extension) and a hcat-mapping csv file

Example:

• fiboa improve file.parquet -o file2.parquet -g -sz -r old=new -pc zstd

Check fiboa improve --help for more details.

Update an extension template with new names

Once you've created and git cloned a new extension, you can use the CLI to update all template placeholders with proper names.

For example, if your extension is meant to have

• the title "Administrative Division Extension",
• the prefix admin (e.g. field admin:country_code or admin:subdivision_code),
• is hosted at https://github.io/vecorel/administrative-division-extension (organization: vecorel, repository /administrative-division-extension),
• and you run Vecorel in the folder of the extension.

Then the following command could be used:

• fiboa rename-extension . -t "Administrative Division" -p admin -s administrative-division-extension -o vecorel

Check fiboa rename-extension --help for more details.

Converter for existing datasets

The CLI ships various converters for existing datasets.

To get a list of available converters/datasets with title, license, etc. run:

• fiboa converters

Use any of the IDs from the list to convert an existing dataset to fiboa:

• fiboa convert de_nrw

See Implement a converter for details about how to

Publish datasets to source coop or your own s3 repository

fiboa publish <dataset> -o <target>

The publish converts and publishes a fiboa dataset to source coop or your own s3 repository. The target directory will be filled with the following files:

<target>/
  <dataset>.parquet
  <dataset>.pmtiles      # requires working ogr2ogr and tippecanoe
  stac/collection.json
  README.md              # generated if --generate-meta/-gm flag is present
  LICENSE.txt            # generated if --generate-meta/-gm flag is present

This directory is synchronized to the s3 repository (default source.coop/fiboa/data).

Requirements: Requires the aws CLI to be installed, and AWS_ACCESS_KEY_ID with AWS_SECRET_ACCESS_KEY environment variables. Also, for generating the pmtiles file, it requires ogr2ogr and tippecanoe.

The command executes the following steps:

• fiboa convert to generate a fiboa parquet dataset. All convert parameters are passed to the converter.
• fiboa validate to validate the fiboa dataset
• creates a .pmtiles from the parquet file. Uses ogr2ogr and tippecanoe
• fiboa create-stac-collection to create a STAC collection
• fiboa publish to publish the fiboa dataset to a source coop or your own s3 repository

Examples:

• fiboa publish at_crop -o data/at_crop
• fiboa publish -c /tmp/cache -gm br_conab -o data/br_conab

Relevant parameters:

• --generate-meta/-gm Generatse the README.md and LICENSE.txt files if absent, based on data-survey and converter properties.
• --data-url The URL to the data repository, used when generating the README
• --s3-upload-path The aws s3 sync target. Defaults to s3://source.coop/fiboa/data . Uploading requires the aws CLI, and AWS_ACCESS_KEY_ID with AWS_SECRET_ACCESS_KEY environment variables.

Check fiboa publish --help for more details.

Development

This project uses Pixi for dependency management and development workflows.

# Install all dependencies including development tools
pixi install -e dev

# Install the package in editable mode
pixi run install-dev

# Run tests
pixi run test

# Format and lint code
pixi run format
pixi run lint

# Run all checks (lint, format, test)
pixi run check

# Install and run pre-commit
pixi run pre-commit-install
pixi run pre-commit-run

Implement a converter

The following high-level description gives an idea how to implement a converter in fiboa CLI:

1. Create a new file in fiboa_cli/datasets based on the template.py
2. Fill in the required variables / test it / run it
3. Add missing dependencies into the appropriate feature group in pixi.toml (or setup.py for pip users)
4. Add the converter to the list above
5. Create a PR to submit your converter for review

Run in Docker

There's a Dockerfile based on a modern Ubuntu+GDAL image. With the Dockerfile, you can run the CLI in a container. This can be handy if you lack the required dependencies on you local machine (e.g. a modern GDAL capable of reading GeoParquet files).

Example usage (mounting /tmp/fiboa as a volume)

docker build . -t fiboa
docker run -it --rm -v /tmp/fiboa:/fiboa fiboa bash

fiboa convert de_nrw -o /fiboa/de_nrw.parquet

If you want to temporarily work on a specific branch of fiboa-cli, you can use the following in the container. This works because the fiboa-cli is pip-installed with -e (in-place):

cd fiboa && git checkout <branch>