DaFab Client

The Python client for DaFab metadata queries, asset downloads and publication.

Install

pip install dafab-client

Python 3.10 or newer is required.

Connection profile

The default account is user_dafab. Set DAFAB_PROFILE before importing the client to select another account. DAFAB_PROFILE_PATH selects an explicit JSON profile. Otherwise, it checks <profile-directory>/<account>/config before the packaged profile. The directory is DAFAB_PROFILE_DIR, then $XDG_CONFIG_HOME/dafab/credentials/profiles when set, or ~/.config/dafab/credentials/profiles.

The profile's ca_cert selects the TLS trust bundle. $profile_path resolves to the directory containing that profile. Keep private credentials outside this repository. Publisher deployment and mounted trust configuration follow the platform runbook.

Quick Start

import dafab_client as dc

print(dc.ping())
print(dc.whoami())
dc.list_storages()

All public helpers are available as dc.<method>.

Copy the packaged simple-user notebook into the current directory.

python -c "import dafab_client as dc; print(dc.get_example('user'))"

Other example profiles are admin, dasi and skim. Copying overwrites existing files with the same names. Operator and publisher notebooks include catalogue writes and should be run cell by cell against the intended profile.

Simple User API Reference

The reference below lists required arguments. Module docstrings describe optional arguments and return values. The enhanced-filter guide covers compound queries.

Session And Discovery

• ping() checks the health of the connected Rucio/STAC service.
• whoami() returns authenticated identity/session details.
• list_stac_scopes() logs visible scopes for the active account.
• list_catalogs_and_collections() logs catalog/collection container ids in the active scope.
• get_catalogs_and_collections() returns structured catalog/collection rows.
• get_items() returns item DID rows.

Filter And Relationship Queries

• get_items_by_enhanced_filter(filter_payload) executes an enhanced STAC filter query.
• get_item_ids_by_collection_field(collection_id) resolves item ids by collection metadata value.
• get_item_ids_by_top_facet_catalog(top_facet_catalog_id) resolves item ids indexed under a top facet catalog.
• get_item_ids_by_facet_value_catalog(facet_value_catalog_id) resolves item ids linked from one facet value catalog.
• get_item_facet_placements(item_id, collection_id=...) returns the facet-value placements currently attached to the Item. Unused collection facets are omitted.
• get_related_item_ids_from_original_item(original_item_id) resolves derived item ids linked via rel=related.
• get_source_original_item_ids_from_derived_item(derived_item_id) resolves original item ids linked via rel=derived_from.
• get_sibling_derived_item_ids(original_item_id) resolves sibling derived items for one original item.

The filter guide explains how inherited=True on comparisons and logical groups queries attachment ancestors. The DaFab server supports enhanced metadata filtering on PostgreSQL and Oracle.

Spatial And Temporal Queries

• get_items_by_timerange(start_date, end_date) filters items by temporal interval.
• get_items_by_bbox(min_long, min_lat, max_long, max_lat) filters items by spatial bounding box.
• get_items_by_bbox_and_timerange(bbox, timerange) filters items by combined bbox and time window.

Metadata Access

• extract_metadata_value(value_path) reads one metadata value path (requires metadata or pname via optional args).
• get_bulk_metadata(pname) retrieves full or partial metadata document values.
• as_json(data) formats payloads for notebook/debug display.

Storage And Asset Operations

• list_storages() logs configured storage endpoints (RSEs).
• check_storage(rse_name) fetches one storage endpoint metadata record.
• list_item_asset_entries(item_id) inspects an item's assets entries.
• build_stable_asset_href(item_id, asset_key) builds canonical stable asset URL used by DaFab STAC metadata.
• download_asset_from_stable_href(stable_href) downloads content by stable asset URL.
• download_item_asset(item_id, asset_key) downloads one metadata asset by key (URL or attached file resolution).
• download_all_derived_item_assets(item_id) downloads all assets for one derived item.

Visualization

• get_map(metadata_batch) renders bbox overlays from metadata to an interactive HTML map.

Item publication and ownership

Each STAC Item is a Rucio CONTAINER carrying its full Feature document. Its own files belong to the directly attached DATASET named item_id + "_assets". The assets dataset has no structured metadata. ensure_item(item_id) creates and verifies this pair. Low-level upload_file calls take the assets dataset name as pname.

The root stac container contains sentinel_2_l2a, which contains original Item containers. Publish and validate the original Item before publishing a generated Item. The original container directly contains its generated containers, which also retain their facet parents. STAC derived_from and related links describe the same lineage. Generated repair can restore this attachment and its facets, but it never creates or repairs the source Item.

Original publication requires TCI_20m. prepare_original_item_metadata(document, asset_paths) renders a thumbnail up to 512 pixels and an overview up to 1,536 pixels, then returns a catalogue copy with the preview assets and stable URLs. The provider document and source raster remain unchanged. ensure_original_item_metadata(item_id, document) initializes an empty Item or updates its controlled preview pair while preserving existing assets and related links. Publish the selected original files and both PNGs through publish_original_asset.

Item asset helpers read only the dedicated assets dataset. They do not traverse generated descendants. An Item deletion plan includes that dataset and its own files and refuses extra children or shared ownership. Run the Item-level dry-run before removing storage objects.

For manual collection checks, call sync_derived_collection_extent(full_scan=True, dry_run=True). It checks facet ownership before reconstructing collection links or extents. With dry_run=False, it removes an extra collection attachment only when the correct owner and its attachment are verified. Missing or conflicting ownership blocks changes to the affected collections. Review the returned errors and facet_ownership_repairs before applying changes.

list_derived_items_with_missing_assets() checks metadata against registered FILE attachments. It also reports empty asset metadata. Its limit caps the number of failing Items returned, not the number scanned. It does not read storage objects or verify their contents.

Local Files And Outputs

Writable helper outputs use DAFAB_DEMO_DATA_DIR when set, otherwise <current working directory>/demo-data. The default map path is <demo-data-dir>/filters/bbox_map.html.

Bundled schemas

Validators use these packaged schemas when no schema path is supplied.

• dasi-original-item.schema.json
• Schema_Copernicus_with_dafab.json
• Schema_DaFab_Facet_Value_Catalog.json
• dafab-smart_agriculture-item.schema.json
• dafab-water_analysis-item.schema.json

Original preflight uses the DASI provider schema before publication. Published originals use Schema_Copernicus_with_dafab.json, which requires the TCI_20m source and its DaFab thumbnail and overview. DASI metadata remains unchanged.

Logging Defaults

dafab_client._rucio.global_utils defaults to debug_mode = False and minimal_logging = True. Minimal logging takes precedence over debug mode.

Releasing

Maintainers should follow docs/releasing.md. Publishing credentials remain in the local private configuration and must never be added to this repository.