sfeos-tools 0.6.0

CLI tools for managing stac-fastapi-elasticsearch-opensearch deployments

CLI tools for managing stac-fastapi-elasticsearch-opensearch deployments.

Table of Contents

• Installation
  • For Elasticsearch
  • For OpenSearch
  • For Viewer
  • For Development
• Usage
• Interactive Demo
• Commands
  • add-bbox-shape
  • reindex
  • load-data
  • ingest-catalog
  • crawl-graph
  • visualize-graph
  • viewer
• Development
• License

Installation

For Elasticsearch

pip install sfeos-tools[elasticsearch]

Or for local development:

pip install -e sfeos_tools[elasticsearch]

For OpenSearch

pip install sfeos-tools[opensearch]

Or for local development:

pip install -e sfeos_tools[opensearch]

For Crawler

To use the crawl-graph command:

pip install sfeos-tools[crawler]

Or for local development:

pip install -e sfeos_tools[crawler]

For Visualizer

To use the visualize-graph command:

pip install sfeos-tools[visualizer]

Or for local development:

pip install -e sfeos_tools[visualizer]

For Viewer

To use the interactive Streamlit viewer:

pip install sfeos-tools[viewer]

Or for local development:

pip install -e sfeos_tools[viewer]

For All Features

To install all optional dependencies (Elasticsearch, OpenSearch, Crawler, Visualizer, and Viewer):

pip install sfeos-tools[all]

Or for local development:

pip install -e sfeos_tools[all]

For Development (both backends)

pip install sfeos-tools[dev]

Or for local development:

pip install -e sfeos_tools[dev]

Usage

After installation, the sfeos-tools command will be available:

# View available commands
sfeos-tools --help

# View version
sfeos-tools --version

Interactive Demo: SKOS to STAC

Want to see the Multi-Tenant Catalogs extension and the ingest-catalog command in action? Check out our interactive Jupyter Notebook tutorial:

📓 SKOS-catalogs-ingestion-demo.ipynb

This notebook walks through a real-world GIS use case (a Traffic Signs taxonomy) and demonstrates:

• Automated Semantic Ingestion: Translating a SKOS RDF file directly into a STAC catalog hierarchy.
• Poly-hierarchy (DAG): Discovering a single spatial asset across multiple departmental catalogs (e.g., Regulatory vs. Warning signs).
• Contextual Breadcrumbs: How SFEOS dynamically rewrites rel="parent" and rel="child" links based on your navigation path.
• Data Safety: Deleting virtual organizational containers without destroying the underlying STAC features.

Commands

Standardized CLI Options

The CLI tools use standardized options across commands for consistency:

Database Commands (add-bbox-shape, reindex):

• --host: Database host (default: localhost or ES_HOST env var)
• --port: Database port (default: 9200 for ES, 9202 for OS, or ES_PORT env var)
• --use-ssl/--no-ssl: SSL connection flag (default: true or ES_USE_SSL env var)
• --user: Database username (default: ES_USER env var)
• --password: Database password (default: ES_PASS env var)
• --api-key: API key for authentication (default: ES_API_KEY env var)

STAC API Commands (load-data, ingest-catalog, viewer):

• --stac-url: STAC API base URL (default: http://localhost:8080)

Authentication Options (optional for STAC API commands):

• --user: Username for basic authentication
• --password: Password for basic authentication
• --api-key: API key for authentication
• --use-ssl/--no-ssl: SSL verification flag

add-bbox-shape

Adds a bbox_shape field to existing collections for spatial search support. This migration is required for collections created before spatial search was added. Collections created or updated after this feature will automatically have the bbox_shape field.

sfeos-tools add-bbox-shape --backend [elasticsearch|opensearch] [options]

Options:

• --backend: Database backend to use (required, choices: elasticsearch, opensearch)
• --host: Database host (default: localhost or ES_HOST env var)
• --port: Database port (default: 9200 for ES, 9202 for OS, or ES_PORT env var)
• --use-ssl/--no-ssl: Use SSL connection (default: true or ES_USE_SSL env var)
• --user: Database username (default: ES_USER env var)
• --password: Database password (default: ES_PASS env var)
• --api-key: API key for authentication (default: ES_API_KEY env var)

Examples:

# Add bbox_shape with default settings
sfeos-tools add-bbox-shape --backend elasticsearch

# Add bbox_shape with custom host and port
sfeos-tools add-bbox-shape --backend elasticsearch --host db.example.com --port 9200

# Add bbox_shape with API key authentication
sfeos-tools add-bbox-shape --backend elasticsearch --api-key your-api-key

# Add bbox_shape with API key and custom host
sfeos-tools add-bbox-shape --backend opensearch --host prod.example.com --api-key your-api-key

reindex

Reindexes all STAC indexes to the next version and updates aliases. This command performs the following actions:

• Creates/updates index templates
• Reindexes collections and item indexes to a new version
• Applies asset migration script for compatibility
• Switches aliases to the new indexes

sfeos-tools reindex --backend [elasticsearch|opensearch] [options]

Options:

• --backend: Database backend to use (required, choices: elasticsearch, opensearch)
• --host: Database host (default: localhost or ES_HOST env var)
• --port: Database port (default: 9200 for ES, 9202 for OS, or ES_PORT env var)
• --use-ssl/--no-ssl: Use SSL connection (default: true or ES_USE_SSL env var)
• --user: Database username (default: ES_USER env var)
• --password: Database password (default: ES_PASS env var)
• --api-key: API key for authentication (default: ES_API_KEY env var)
• --yes: Skip confirmation prompt

Examples:

# Reindex Elasticsearch with custom host and no SSL
sfeos-tools reindex --backend elasticsearch --host localhost --port 9200 --no-ssl --yes

# Reindex OpenSearch with default settings
sfeos-tools reindex --backend opensearch --yes

# Reindex with API key authentication
sfeos-tools reindex --backend elasticsearch --api-key your-api-key --yes

# Reindex with API key and custom host
sfeos-tools reindex --backend opensearch --host prod.example.com --api-key your-api-key --yes

load-data

Load STAC collections and items from local JSON files into a STAC API instance. This command is useful for:

• Populating a new STAC API deployment with test data
• Migrating data between STAC API instances
• Bulk loading STAC collections and items

sfeos-tools load-data --stac-url <stac-api-url> [options]

Options:

• --stac-url: STAC API base URL (default: http://localhost:8080)
• --collection-id: ID of the collection to create/update (default: test-collection)
• --data-dir: Directory containing collection.json and feature collection files (default: sample_data/)
• --use-bulk: Use bulk insert method for items (faster for large datasets)

Data Directory Structure:

Your data directory should contain:

• collection.json: STAC collection definition
• One or more .json files: Feature collections with STAC items

Examples:

# Load data from default directory
sfeos-tools load-data --stac-url http://localhost:8080

# Load with custom collection ID and bulk insert
sfeos-tools load-data \
  --stac-url http://localhost:8080 \
  --collection-id my-collection \
  --use-bulk

# Load from custom directory
sfeos-tools load-data \
  --stac-url http://localhost:8080 \
  --data-dir /path/to/stac/data \
  --collection-id production-data

ingest-catalog

Note: to enable this functionality, the ENABLE_CATALOGS_ROUTE env var needs to be set in SFEOS.

Ingest SKOS/RDF-XML files to create STAC catalogs and sub-catalogs. This command parses RDF/XML files containing SKOS concepts and creates a hierarchical catalog structure in the STAC API. It handles:

• Creating catalogs from SKOS concepts
• Establishing parent-child relationships (skos:narrower)
• Preserving semantic links (skos:related, skos:exactMatch, etc.)
• Including metadata from definitions and modification dates

sfeos-tools ingest-catalog --xml-file <path-to-rdf-xml> [options]

Options:

• --xml-file: Path to RDF/XML file containing SKOS concepts (required)
• --stac-url: STAC API base URL (default: http://localhost:8080)
• --user: Username for basic authentication (optional)
• --password: Password for basic authentication (optional)
• --api-key: API key for authentication (optional)
• --use-ssl/--no-ssl: Enable or disable SSL verification (optional)

Examples:

# Ingest test data from the tests directory (uses default localhost:8080)
sfeos-tools ingest-catalog --xml-file tests/skos-test-topics.rdf

# Ingest with explicit STAC API URL
sfeos-tools ingest-catalog --xml-file demo-notebooks/traffic-signs.rdf --stac-url http://localhost:8080

# Ingest with basic authentication
sfeos-tools ingest-catalog --xml-file concepts.xml --stac-url https://my-stac-api.com --user myuser --password mypass

# Ingest with API key authentication
sfeos-tools ingest-catalog --xml-file concepts.xml --stac-url https://my-stac-api.com --api-key your-api-key

# Ingest with SSL verification disabled
sfeos-tools ingest-catalog --xml-file concepts.xml --stac-url https://my-stac-api.com --no-ssl

# Ingest with API key and custom SSL settings
sfeos-tools ingest-catalog --xml-file /path/to/concepts.xml --stac-url https://my-stac-api.com --api-key your-api-key --no-ssl

crawl-graph

Crawl the SFEOS Multi-Tenant Catalogs and Collections to build and display the Directed Acyclic Graph (DAG) of your SFEOS instance. This command traverses the entire catalog hierarchy using breadth-first search, discovering all catalogs and collections regardless of whether the backend has a dedicated graph endpoint installed.

Key Features:

• Discovers all catalogs and collections in your SFEOS instance
• Works with any SFEOS deployment (no special endpoints required)
• Prevents infinite loops in poly-hierarchy DAGs using a visited set
• Two output formats: hierarchical text tree or JSON graph data
• Progress tracking with visual progress bar
• Graceful error handling for connection failures

sfeos-tools crawl-graph [options]

Options:

• --url: Base URL of the SFEOS API (default: http://localhost:8080)
• --output: Output format: text for hierarchical tree view or json for graph data (default: text)

Requirements:

The crawler requires networkx. Install with:

pip install sfeos-tools[crawler]

Examples:

# Crawl default localhost instance with text output
sfeos-tools crawl-graph

# Crawl a custom SFEOS instance
sfeos-tools crawl-graph --url https://my-sfeos-api.com

# Get JSON output for programmatic use
sfeos-tools crawl-graph --url http://localhost:8080 --output json

# Crawl custom instance and output as JSON
sfeos-tools crawl-graph --url https://my-sfeos-api.com --output json

Example Output (Text Format):

🌲 Crawling SFEOS Multi-Tenant API at http://localhost:8080...
Traversing Hierarchy  [####################################]  100%
✅ Crawl complete! Discovered 13 entities.

--- SFEOS Topology ---
  └─ 📁 tenant-a
    └─ 📁 forestry-dept
      └─ 📄 sentinel-2-l2a
  └─ 📁 tenant-b
    └─ 📄 landsat-8-c2-l2

The JSON output provides a complete graph representation suitable for analysis, visualization, or integration with other tools.

visualize-graph

Generate an interactive, physics-simulated web visualization of your SFEOS catalog hierarchy. This command crawls the API and renders a beautiful, drag-and-drop HTML dashboard showing the complete DAG with color-coded nodes and automatic browser launch.

Key Features:

• Physics-simulated hierarchical tree layout with spring forces
• Color-coded nodes for quick visual understanding:
  • 🔵 Blue: Standard catalogs
  • 🟢 Green: Leaf catalogs (no children)
  • 🟣 Purple Boxes: Collections (leaf nodes in STAC hierarchy)
  • 🟠 Orange Diamonds: Poly-hierarchical nodes (multiple parents)
  • 🔴 Red: Virtual root API node
• Interactive drag-and-drop node manipulation
• Hover tooltips showing catalog/collection IDs and metadata
• Dark mode UI for comfortable viewing
• Automatic browser launch with generated HTML file
• Works with any SFEOS deployment (no special endpoints required)

sfeos-tools visualize-graph [options]

Options:

• --url: Base URL of the SFEOS API (default: http://localhost:8080)
• --layout: Graph layout style (default: hierarchical)
  • hierarchical: Tree layout flowing top-to-bottom (best for DAGs)
  • hierarchical-lr: Tree layout flowing left-to-right (alternative hierarchical view)
  • force: Force-directed layout using ForceAtlas2 algorithm (organic, physics-based)
  • spring: Spring-based layout using Barnes-Hut algorithm (natural node spreading)

Requirements:

The visualizer requires networkx and pyvis. Install with:

pip install sfeos-tools[visualizer]

Examples:

# Visualize default localhost instance (hierarchical layout)
sfeos-tools visualize-graph

# Visualize with left-to-right hierarchical layout
sfeos-tools visualize-graph --layout hierarchical-lr

# Visualize with force-directed layout
sfeos-tools visualize-graph --layout force

# Visualize with spring layout
sfeos-tools visualize-graph --layout spring

# Visualize a custom SFEOS instance with hierarchical layout
sfeos-tools visualize-graph --url https://my-sfeos-api.com

# Visualize custom instance with left-to-right layout
sfeos-tools visualize-graph --url https://my-sfeos-api.com --layout hierarchical-lr

Output:

The command generates an sfeos_topology.html file and automatically opens it in your default browser. The visualization features:

• Hierarchical tree layout flowing top-to-bottom
• Nodes that can be dragged to explore relationships
• Spring physics that naturally separates nodes
• Poly-hierarchical nodes highlighted as orange diamonds with multiple incoming arrows
• Responsive design that fills your entire browser window

viewer

Launch an interactive Streamlit-based web viewer for exploring STAC collections and items. The viewer provides:

• Interactive map visualization of STAC items
• Collection browser and selector
• Item search and filtering
• Metadata inspection
• Asset preview and imagery display
• Support for thumbnails, images (JPEG, PNG, TIFF), and other asset types

sfeos-tools viewer [options]

Options:

• --stac-url: STAC API base URL (default: http://localhost:8080)
• --port: Port for the Streamlit viewer (default: 8501)

Requirements:

The viewer requires additional dependencies. Install with:

pip install sfeos-tools[viewer]

Examples:

# Launch viewer with default settings (connects to http://localhost:8080)
sfeos-tools viewer

# Connect to a custom STAC API
sfeos-tools viewer --stac-url https://my-stac-api.com

# Use a different port
sfeos-tools viewer --port 8502

# Custom STAC API and port
sfeos-tools viewer --stac-url http://localhost:8080 --port 8502

The viewer will automatically open in your default web browser. Press Ctrl+C in the terminal to stop the viewer.

Development

To develop sfeos-tools locally:

# Install in editable mode with dev dependencies
pip install -e ./sfeos_tools[dev]

# Run the CLI
sfeos-tools --help

# Run tests
pytest

# Format code
pre-commit install
pre-commit run --all-files

License

This project is licensed under the MIT License - see the LICENSE file for details.