fiboa CLI
A command-line interface (CLI) for working with fiboa files.
Getting Started
In order to make working with fiboa easier we have developed command-line interface (CLI) tools such as inspection, validation and file format conversions.
Installation
Using Pixi (Recommended)
This project uses Pixi for dependency management. Install Pixi first, then:
# Clone the repository and navigate to it
git clone https://github.com/fiboa/cli.git
cd cli
# Install all dependencies
pixi install
# Run the CLI
pixi run fiboa
Using pip
Alternatively, you can install from PyPI with Python 3.11 or any later version:
pip install fiboa-cli
Execute a command
After the installation you should be able to run the following command: fiboa (or pixi run fiboa if using Pixi)
You should see usage instructions and available commands for the CLI.
fiboa CLI supports various commands to work with the files:
- fiboa CLI
- Getting Started
- Commands
- Validation
- Create fiboa GeoParquet from GeoJSON
- Create fiboa GeoJSON from GeoParquet
- Inspect fiboa GeoParquet file
- Merge fiboa GeoParquet files
- Create JSON Schema from fiboa Schema
- Validate a Vecorel Schema
- Improve a fiboa Parquet file
- Update an extension template with new names
- Converter for existing datasets
- Publishing with Portolan
- Development
- Run in Docker
Commands
Validation
To validate a fiboa GeoParquet or GeoJSON file, you can for example run:
- GeoJSON:
fiboa validate example.json --collection collection.json - GeoParquet:
fiboa validate example.parquet --data
Check fiboa validate --help for more details.
The validator also supports remote files.
http://orhttps://: no further configuration is needed.s3://: With Pixi, runpixi install -e s3or with pip, runpip install fiboa-cli[s3]and you may need to set environment variables. Refer to the s3fs credentials documentation for how to define credentials.gs://: With Pixi, runpixi install -e gcsor with pip, runpip install fiboa-cli[gcs]. By default,gcsfswill attempt to use your default gcloud credentials or, attempt to get credentials from the google metadata service, or fall back to anonymous access.
Create fiboa GeoParquet from GeoJSON
To create a fiboa-compliant GeoParquet for a fiboa-compliant set of GeoJSON files containing Features or FeatureCollections, you can for example run:
fiboa create-geoparquet geojson/example.json -o example.parquet -c geojson/collection.json
Check fiboa create-geoparquet --help for more details.
Create fiboa GeoJSON from GeoParquet
To create one or multiple fiboa-compliant GeoJSON file(s) for a fiboa-compliant GeoParquet file, you can for example run:
-
GeoJSON FeatureCollection:
fiboa create-geojson example.parquet -o dest-folder -
GeoJSON Features (with indentation and max. 100 features):
fiboa create-geojson example.parquet -o dest-folder -n 100 -i 2 -ffiboa create-geojson example.parquet -o dest-folder -n 100 -i 2 -f
Check fiboa create-geojson --help for more details.
Inspect fiboa GeoParquet file
To look into a fiboa GeoParquet file to get a rough understanding of the content, the following can be executed:
fiboa describe example.parquet
Check fiboa describe --help for more details.
Merge fiboa GeoParquet files
Merges multiple fiboa datasets to a combined fiboa dataset:
fiboa merge ec_ee.parquet ec_lv.parquet -o merged.parquet -e https://fiboa.org/hcat-extension/v0.1.0/schema.yaml -i ec:hcat_name -i ec:hcat_code -i ec:translated_name
Check fiboa merge --help for more details.
Create JSON Schema from fiboa Schema
To create a JSON Schema for a fiboa Schema YAML file, you can for example run:
fiboa jsonschema example.json --id=https://vecorel.org/specification/v0.3.0/geojson/schema.json -o schema.json
Check fiboa jsonschema --help for more details.
Validate a Vecorel Schema
To validate a Vecorel Schema YAML file, you can for example run:
fiboa validate-schema schema/schema.yaml
Check fiboa validate-schema --help for more details.
Improve a fiboa Parquet file
Various "improvements" can be applied to a fiboa GeoParquet file. The commands allows to
- detect fiboa-0.2 files and convert them to fiboa-0.3
- change the CRS (
--crs) - change the GeoParquet version (
-gp1) and compression (-pc) - add/fill missing perimeter/area values (
-sz) - fix invalid geometries (
-g) - rename columns (
-r) - add HCAT columns (
--hcat=mapping.csv) based on the input crop:code (from crop-extension) and a hcat-mapping csv file
Example:
fiboa improve file.parquet -o file2.parquet -g -sz -r old=new -pc zstd
Check fiboa improve --help for more details.
Update an extension template with new names
Once you've created and git cloned a new extension, you can use the CLI to update all template placeholders with proper names.
For example, if your extension is meant to have
- the title "Administrative Division Extension",
- the prefix
admin(e.g. fieldadmin:country_codeoradmin:subdivision_code), - is hosted at
https://github.io/vecorel/administrative-division-extension(organization:vecorel, repository/administrative-division-extension), - and you run Vecorel in the folder of the extension.
Then the following command could be used:
fiboa rename-extension . -t "Administrative Division" -p admin -s administrative-division-extension -o vecorel
Check fiboa rename-extension --help for more details.
Converter for existing datasets
The CLI ships various converters for existing datasets.
To get a list of available converters/datasets with title, license, etc. run:
fiboa converters
Use any of the IDs from the list to convert an existing dataset to fiboa:
fiboa convert de_nrw
See Implement a converter for details about how to
Publishing with Portolan
fiboa datasets are published as a Portolan catalog. The fiboa CLI converts and validates; Portolan writes the STAC metadata, the PMTiles, the checksums and the README, and uploads. The steps below are written so that an agent can follow them; the Portolan skills cover the Portolan side in more depth.
Inside a Portolan catalog (portolan init), for dataset <id> and edition <variant>:
- Convert and validate. Name the file after the edition, so each edition is its own asset:
fiboa convert <id> --variant <variant> -c <cache> -o <id>/<id>-<variant>.parquet fiboa validate <id>/<id>-<variant>.parquet
- Seed the collection metadata from the converter: title, description, providers, license, fiboa version, and the
columns with a description for every fiboa property. The columns move to the collection; the assets go, as
Portolan adds its own. Only for a new collection; Portolan keeps these fields afterwards.
fiboa create-stac-collection <id>/<id>-<variant>.parquet -o <id>/stac.json jq '."table:columns" = .assets.data."table:columns" | del(.assets)' <id>/stac.json > <id>/collection.json rm <id>/stac.json
- Add the file with its campaign date, and generate the PMTiles (requires
tippecanoe):
portolan add <id> --datetime <variant>-01-01 --pmtiles
- Describe the dataset from its data survey: usually
<ID>.mdwith the id upper-cased and_as-(de_nrwisDE-NRW.md), otherwise the country's file (nl_blockis inNL.md).-
Run
portolan metadata init <id>and fill<id>/.portolan/metadata.yaml:data survey metadata.yaml Data Provider (Legal Entity) providers, roleproducer(andlicensor)Homepage, Data URL source_urlLicense license,license_urlOverview and the dataset's section descriptionCaveats in the text (coverage, preliminary editions) known_issuesthe fiboa project, publishing this copy contact, andproviderswith rolehost -
Describe the dataset's own columns in
<id>/collection.json(table:columns[].description) from the survey's Properties table; the converter'scolumnsshow which source column each one comes from. -
Run
portolan readme <id>.
-
- Complete what else Portolan asks for until
portolan checkpasses, such as a thumbnail (skillportolan-thumbnails). - Upload:
portolan push <remote> --collection <id>(skillsourcecoopfor Source Cooperative).
Development
This project uses Pixi for dependency management and development workflows.
# Install all dependencies including development tools
pixi install -e dev
# Install the package in editable mode
pixi run install-dev
# Run tests
pixi run test
# Format and lint code
pixi run format
pixi run lint
# Run all checks (lint, format, test)
pixi run check
# Install and run pre-commit
pixi run pre-commit-install
pixi run pre-commit-run
Implement a converter
The following high-level description gives an idea how to implement a converter in fiboa CLI:
- Create a new file in
fiboa_cli/datasetsbased on thetemplate.py - Fill in the required variables / test it / run it
- Add missing dependencies into the appropriate feature group in
pixi.toml(orsetup.pyfor pip users) - Add the converter to the list above
- Create a PR to submit your converter for review
Run in Docker
There's a Dockerfile based on a modern Ubuntu+GDAL image. With the Dockerfile, you can run the CLI in a container. This can be handy if you lack the required dependencies on you local machine (e.g. a modern GDAL capable of reading GeoParquet files).
Example usage (mounting /tmp/fiboa as a volume)
docker build . -t fiboa
docker run -it --rm -v /tmp/fiboa:/fiboa fiboa bash
fiboa convert de_nrw -o /fiboa/de_nrw.parquet
If you want to temporarily work on a specific branch of fiboa-cli, you can use the following in the container.
This works because the fiboa-cli is pip-installed with -e (in-place):
cd fiboa && git checkout <branch>
Metadata
Release files for fiboa-cli 0.30.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| fiboa_cli-0.30.0.tar.gz | 156.4 kB | Without provenance | Details for fiboa_cli-0.30.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| fiboa_cli-0.30.0-py3-none-any.whl | Python 3 | none | any | Details for fiboa_cli-0.30.0-py3-none-any.whl |
Total release size: 342.5 kB
Release files / fiboa_cli-0.30.0.tar.gz
| Download URL | fiboa_cli-0.30.0.tar.gz |
|---|---|
| Size | 156.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
10ca345177c09ea76a4f9089f34233b79af48eb06a3f9cfe2366dc0737e8ae1b
|
|
BLAKE2b-256 checksum How to use checksums |
f8a6dcaffca19ec5153f30320182e99565b1a78c7a0c57be079b99cd273790a6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.9.25
|
Release files / fiboa_cli-0.30.0-py3-none-any.whl
| Download URL | fiboa_cli-0.30.0-py3-none-any.whl |
|---|---|
| Size | 186.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2534d3017e09c0144f28e65d31480c867d5a3e89a8e0e4ba0a9d9cc02e7257d7
|
|
BLAKE2b-256 checksum How to use checksums |
b50caae60d2d93e091e551e714c3d48115ded024d4b048a5d83e82dacc4f0435
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.9.25
|
Security for fiboa-cli 0.30.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.