User Generated Data Process

Table of Contents:

1. Overview
2. User Workflow
3. Required Structure
4. Metadata Files
   • collection.json
   • collection_config.json
5. Generate Item Metadata
6. Regenerate Item Metadata in S3
7. Maintainer Notes
8. Optional S3 Upload
9. Runtime Initialization
10. Troubleshooting
11. External References

This project helps you prepare a User Generated collection for DestinE Data Lake review.

Use this guide to:

1. Structure your collection folders and data assets.
2. Configure metadata files.
3. Generate STAC Item metadata.

For contributor and maintenance documentation, see docs/USERGENERATED_DEVELOPMENT.md.

For a feature overview (data provider and operations team capabilities), see docs/USERGENERATED_FEATURES.md.

For an overview of how the package is structured and how it interacts with S3 and the deployed STAC API, see docs/USERGENERATED_HLA.md.

For a log of notable changes by version, see docs/USERGENERATED_RELEASE_NOTES.md.

User Workflow

1. Create your collection root folder using the provided collection id, for example EO.XXX.YYY.ZZZ.
2. If needed, place the collection in a subfolder under your current working directory and pass that subfolder with --collection-path.
3. Add metadata files under metadata/.
4. Organize data assets under data/ according to your configured folder level.
5. Run generate_item_metadata.py to validate and generate STAC Items under metadata/items/.

If you mirror a collection from S3 before regenerating items, mirrored data folders may contain usergenerated_s3_folder_manifest.jsonl files instead of placeholder dummy assets. Item generation detects these manifests automatically and uses their entries as the authoritative asset list.

Reference collection structure is available in EO.XXX.YYY.ZZZ.

Required Structure

Expected top-level structure:

EO.XXX.YYY.ZZZ/
  data/
  metadata/
    collection.json
    collection_config.json
    items/

Notes:

1. Collection ids are case-sensitive and typically uppercase with . separators.
2. The metadata/items/ folder is where generated Item JSON files are written.

Metadata Files

collection.json

How to Complete collection.json

Use metadata/collection.json as the authoritative STAC Collection metadata file for your dataset.

---

Required STAC Collection fields

The following fields must be completed and valid:

1. type — Must be set to Collection.
2. stac_version — Use the current version (1.1.0). generate-item-metadata rejects the collection if stac_version is missing or set to anything else.
3. id — Your collection identifier. This must match:
   • your root folder name
   • collection_config.json > id
4. title — A human-readable name for your collection.
5. description — A detailed description of the dataset.
6. dedl:short_description — A concise version of the description. Displayed in the DestinE DataLake UI.
7. license — The license under which the data is distributed.
8. extent — Must include both spatial and temporal components.
9. links — Must be present as an array.
10. assets — Include if you have collection-level assets; otherwise use {}.

---

Recommended best practices

• Keep the id consistent across:
  • the collection root folder name
  • collection_config.json > id
• Use a clear title and a meaningful description that explains:
  • dataset content
  • data source(s)
  • intended use cases
• Ensure extent is accurate so item-level metadata is interpreted correctly.
• Only update necessary links (typically describedby and cite-as).
• Keep type and stac_version unchanged unless upgrading STAC versions.

---

Field-specific guidance

links

• Always include at least:
  • license → link to license terms
  • cite-as → DOI or citation reference
  • describedby → external dataset documentation
• Ensure links are stable and publicly accessible.
• Use meaningful title values so users understand what each link represents.

---

extent

• Spatial (bbox):

  • Format: [west, south, east, north]
  • Use WGS84 coordinates (EPSG:4326).
  • Ensure the bounding box fully covers your dataset.

• Temporal (interval):

  • Use ISO 8601 format (YYYY-MM-DDTHH:MM:SSZ).
  • Use:
    • a closed interval [start, end] for fixed datasets
    • [start, null] for ongoing datasets

---

license

• Prefer SPDX identifiers (e.g. CC-BY-4.0, MIT, Apache-2.0).
• Use:
  • proprietary for restricted datasets
  • various if multiple licenses apply
• If using proprietary or various:
  • explain clearly in description
  • provide license links in links

---

providers

• List all organizations involved in the dataset lifecycle.
• Common roles:
  • producer → created the data
  • processor → processed or transformed the data
  • licensor → owns or distributes the data rights
  • host → (optional) hosting platform
• Provide:
  • name (required)
  • url (recommended)
• Include multiple providers if applicable rather than combining roles into one.

---

sci:publications

• Use this field to reference scientific publications or datasets.
• Include:
  • sci:doi → DOI identifier
  • sci:citation → human-readable citation
• Add multiple entries if there are several related publications.
• Ensure DOIs resolve correctly (test the links).

---

keywords

• Provide relevant, searchable terms describing the dataset.
• Use:
  • domain-specific terminology (e.g. Land Cover, Atmospheric Data)
  • common synonyms where helpful
• Avoid overly generic terms (e.g. “data”, “analysis”).

---

assets

• Use for collection-level resources (not item-level data).
• Common examples:
  • thumbnail → preview image
  • overview → documentation PDF
• Include:
  • href (URL)
  • type (MIME type, e.g. image/png)
  • roles (e.g. thumbnail, overview)
• If no assets are needed, use an empty object:

  "assets": {}
  

collection_config.json

This controls how Item metadata generation behaves. Base example in EO.XXX.YYY.ZZZ/metadata/collection_config.json.

Configuration Options

The following options are user-facing and supported during item generation:

Key	Type	Default	Description
id	string	required	Collection id. Must match your collection root folder id.
item_folder_level	string	DD	Where item folders are discovered. Allowed values: YYYY, MM, DD, NONE.
item_config_optional	boolean	false	If false, each item folder is expected to contain item_config.json.
item_asset_ignore_list	list[string]	[]	File names to ignore when creating STAC assets. An Item whose files are all ignored has no assets and fails generation.
media_type_overrides	object	{}	Optional extension to MIME type map used before default detection, e.g. { ".grib2": "application/x-grib2" }.
role_overrides	object	{}	Optional MIME type to role map. Overrides default data/metadata classification. Allowed roles: data, metadata, thumbnail, overview. E.g. { "application/x-grib2": "data" }. File-name regex patterns (thumbnail_regex, overview_regex) still take priority.
thumbnail_regex	string	unset	Regex used to detect thumbnail assets and assign thumbnail role.
overview_regex	string	unset	Regex used to detect overview assets and assign overview role.
additional_property_keys	list[string]	[]	Additional property suffix keys parsed from item folder names (standard mode).
item_folder_naming_convention_type	string	standard	Naming convention mode. Allowed values: standard, non_standard_1.
properties	object	{}	Additional properties merged into each generated Item.
bbox	list[number]	unset	Bounding box [minx, miny, maxx, maxy] applied to generated items.
item_date_overide	string	unset	Simplified mode override date in YYYYMMDD format.
asset_href_prefix	string	unset	Optional prefix prepended to every generated asset href, e.g. a public bucket URL such as https://s3.central.data.destination-earth.eu/. A trailing slash is normalized; an empty string behaves as unset. Can be overridden at run time — see Overriding the prefix at run time.

Important: the accepted key name is item_date_overide (spelling as shown).

Folder Granularity Options

Set item_folder_level in collection_config.json.

1. DD: item folders are expected under data/YYYY/MM/DD/<item_folder>.
2. MM: item folders are expected under data/YYYY/MM/<item_folder>.
3. YYYY: item folders are expected under data/YYYY/<item_folder>.
4. NONE: simplified mode. Any folder directly under data/ becomes an Item.

Structure Conformance Check

Before any Item is written, generate-item-metadata checks every file under data/ against the configured item_folder_level and fails if anything does not fit. It also detects Item folders that contain no files at all — an empty folder would otherwise silently produce a STAC Item with zero assets. All problems are reported in a single run, so a mis-structured collection can be fixed in one pass rather than one error at a time.

Nothing is written to metadata/items/ when the check fails.

Reported reason	What it means	How to fix it
empty_item_folder	An Item folder contains no files (not even in nested subfolders).	Remove the folder, or add the Item's asset files to it.
loose_file	A file sits above item level, e.g. directly in data/2024/11/15/. It belongs to no Item, so it would never become an asset.	Move it into an item folder, or remove it.
invalid_year_folder	A folder under data/ is not a 4-digit year.	Rename it, e.g. 2024.
invalid_month_folder	A month folder is not a zero-padded 01-12.	Rename it, e.g. 01, not 1.
invalid_day_folder	A day folder is not a zero-padded 01-31.	Rename it, e.g. 05, not 5.
invalid_calendar_date	The YYYY/MM/DD folders are well-formed but not a real date, e.g. 2024/02/30.	Correct the date folders.
invalid_item_folder_name	The item folder name cannot be parsed as an item id: it does not start with the collection id, its datetime segment is malformed, or its __ suffixes do not match additional_property_keys.	Rename it to <collection_id>_<start_datetime>[_<end_datetime>], see Standard Mode Example.

A folder is reported once however many files sit under it, and reporting stops after 50 findings.

The same check runs during regenerate-item-metadata-s3, against the local mirror of the bucket. Mirror bookkeeping files (usergenerated_s3_folder_manifest.jsonl) are not collection content and are never reported: the objects they describe are checked under their real names, so a reported path is always one that exists in your bucket. An empty Item folder in S3 is reported too: a folder created in the S3 console (or by tools such as s3cmd) is a zero-byte marker object whose key ends in /, and the mirror recreates it as an empty local folder. A marker on a folder that also holds objects is not reported.

Separately, every Item needs at least one real asset. If all files in an Item folder are excluded by item_asset_ignore_list (e.g. it holds only item_config.json), generation stops with Item folder 'data/...' has no assets. Remove the folder or add the Item's asset files. Unlike the checks above, this is raised for the first offending Item rather than listing all of them.

When item_folder_level is NONE, simplified mode imposes no layout, so every folder directly under data/ is treated as an Item; the layout checks are skipped, but empty Item folders are still reported.

Standard Mode Example

{
  "id": "EO.XXX.YYY.ZZZ",
  "item_asset_ignore_list": ["item_config.json"],
  "media_type_overrides": {
    ".grib2": "application/x-grib2"
  },
  "item_config_optional": false,
  "item_folder_level": "DD",
  "thumbnail_regex": "^thumbnail",
  "overview_regex": "^overview",
  "additional_property_keys": []
}

Simplified Mode Example

Simplified mode should only be used with agreement from support.

{
  "id": "EO.XXX.YYY.ZZZ",
  "item_asset_ignore_list": ["item_config.json"],
  "item_folder_level": "NONE",
  "item_date_overide": "20240101",
  "thumbnail_regex": "^thumbnail",
  "overview_regex": "^overview",
  "additional_property_keys": [],
  "bbox": [-10.0, 35.0, 10.0, 60.0]
}

Simplified mode datetime behavior:

1. If item_date_overide is set, that date is used for all generated Items.
2. Otherwise, if the first folder level under data/ is a year, YYYY-01-01 is used.
3. Otherwise, the generation date is used.

Asset Href Prefix

By default, generated asset hrefs are relative to the collection root, e.g.:

"href": "data/2024/11/15/EO.XXX.YYY.ZZZ_20241115T000000_20241115T235959/20241115.png"

Set asset_href_prefix in collection_config.json to prepend a fixed prefix (typically a public bucket URL) to every generated asset href:

{
  "id": "EO.XXX.YYY.ZZZ",
  "asset_href_prefix": "https://s3.central.data.destination-earth.eu/"
}

which produces:

"href": "https://s3.central.data.destination-earth.eu/data/2024/11/15/EO.XXX.YYY.ZZZ_20241115T000000_20241115T235959/20241115.png"

The prefix applies to every generated asset regardless of role (data, metadata, thumbnail, overview); item/collection self links are unaffected. This works identically for generate-item-metadata, regenerate-item-metadata-s3, and promote_collection — whichever collection_config.json is in effect for the collection being processed is the one consulted.

Overriding the prefix at run time

Every entry point also accepts an explicit prefix that overrides any asset_href_prefix set in collection_config.json or in a per-item item_config.json. No config file is modified — in S3 the source collection_config.json stays as it is; only the generated item hrefs change.

Entry point	Override
generate-item-metadata	--asset-href-prefix URL
ItemGenerator(...)	asset_href_prefix_override="URL"
regenerate-item-metadata-s3	--asset-href-prefix URL
ItemGeneratorS3(...)	asset_href_prefix_override="URL"
promote_collection(...)	asset_href_prefix="URL"
script_s3_promote_collection.py / script_s3_regenerate_item_metadata.py	S3_ASSET_HREF_PREFIX env var

• Omitted / None (or the env var unset): no override, the configured value is used as before.
• An empty string (--asset-href-prefix "", S3_ASSET_HREF_PREFIX=): forces no prefix, even when one is configured.
• The same trailing-slash normalization applies.

promote_collection(
    collection_id="EO.XXX.YYY.ZZZ",
    ...,
    on_ready=on_ready,
    asset_href_prefix="https://s3.central.data.destination-earth.eu/my-target-bucket/EO.XXX.YYY.ZZZ/",
)

Per-Item Configuration

Each item folder can include an item_config.json file. Typical per-item overrides:

1. bbox
2. properties
3. Asset-related regex fields when needed

When present, item-level values override collection-level values.

Generate Item Metadata

CLI Usage

Primary command (after installing the package):

generate-item-metadata EO.XXX.YYY.ZZZ

This default expects the collection at ./EO.XXX.YYY.ZZZ.

Optional collection subfolder path:

generate-item-metadata EO.XXX.YYY.ZZZ --collection-path toto

With this option, the collection is resolved at ./toto/EO.XXX.YYY.ZZZ. For safety, --collection-path is currently restricted to subfolders under the current working directory.

Optional bucket override:

generate-item-metadata EO.XXX.YYY.ZZZ --bucket-name usergenerated-proposal-eo.xxx.yyy.zzz

Optional asset href prefix override (outranks any configured asset_href_prefix; "" forces no prefix — see Overriding the prefix at run time):

generate-item-metadata EO.XXX.YYY.ZZZ --asset-href-prefix "https://s3.central.data.destination-earth.eu/my-bucket/EO.XXX.YYY.ZZZ/"

Compatibility command (run from the project root):

python generate_item_metadata.py EO.XXX.YYY.ZZZ

Compatibility command with collection subfolder:

python generate_item_metadata.py EO.XXX.YYY.ZZZ --collection-path toto

Programmatic Usage

You can also use ItemGenerator directly in your Python scripts:

from usergenerated import ItemGenerator

# Create a generator for your collection
generator = ItemGenerator("EO.XXX.YYY.ZZZ")

# Generate Item metadata
generator.run()

# Resolve collection from ./toto/EO.XXX.YYY.ZZZ
generator_with_path = ItemGenerator("EO.XXX.YYY.ZZZ", collection_path="toto")
generator_with_path.run()

# With optional bucket name override
generator_with_bucket = ItemGenerator("EO.XXX.YYY.ZZZ", override_bucket_name="custom-bucket")
generator_with_bucket.run()

# Also write a reordered/normalised copy to output/collection_reordered.json (off by default)
generator_reordered = ItemGenerator("EO.XXX.YYY.ZZZ", save_reordered_collection=True)
generator_reordered.run()

Generated output is written to:

<COLLECTION_ID>/metadata/items/*.json

The ItemGenerator class is the primary API for programmatic use. The CLI command is a thin wrapper around this class.

Regenerate Item Metadata in S3

Use ItemGeneratorS3 to regenerate STAC Item metadata from bucket data while restricting updates to Item metadata only. This is useful when you need to update item generation logic or fix generated metadata without modifying data files in S3.

CLI Usage

Primary command (regenerate items in S3 for your collection):

regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ

This command:

1. Checks the delivery follows the standard (strict structure mode, on by default), then mirrors collection data from S3 to a local folder.
2. Regenerates STAC Item metadata locally with your configured settings — the result is exactly the items backed by a folder under data/, so an item whose data folder has been removed or renamed is dropped.
3. Backs up existing S3 items by default.
4. Replaces only the metadata/items prefix in S3 (never modifies data files).

Optional dry-run (validate without modifying S3):

regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --dry-run

Optional bucket name override (a non-standard name requires turning strict structure mode off):

regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --bucket-name custom-bucket-name --no-strict-structure

Optional local mirror folder (default: tmp_structure_mirror):

regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --local-root "./my_mirror"

Disable backup:

regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --no-backup-items

Skip the ready_for_transfer safety check (see Safety Behavior; enabled by default):

regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --no-use-ready-for-transfer-flag

Override the asset href prefix for the regenerated items (the collection_config.json in S3 is not modified; "" forces no prefix):

regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --asset-href-prefix "https://s3.central.data.destination-earth.eu/my-bucket/EO.XXX.YYY.ZZZ/"

Demonstration script usage (env-first configuration via .env/environment variables, see script_s3_regenerate_item_metadata.py for the full list of S3_* variables):

python script_s3_regenerate_item_metadata.py

This script is intentionally a thin example of programmatic ItemGeneratorS3 usage. For routine operations, prefer the regenerate-item-metadata-s3 console command.

The demo script additionally supports USE_READY_FOR_TRANSFER_FLAG (or the use_ready_for_transfer_flag keyword argument to run_regenerate_item_metadata) to toggle the ready_for_transfer safety check described below — see Safety Behavior. The primary console command exposes the same toggle via --use-ready-for-transfer-flag/--no-use-ready-for-transfer-flag (shown above). Both also accept an asset href prefix override (S3_ASSET_HREF_PREFIX / --asset-href-prefix) — see Overriding the prefix at run time — and a strict structure mode toggle (S3_STRICT_STRUCTURE / --no-strict-structure) — see Strict structure mode.

Programmatic Usage

You can also use ItemGeneratorS3 directly in your Python scripts:

from usergenerated import ItemGeneratorS3

# Create a generator for your collection
generator = ItemGeneratorS3(
    "EO.XXX.YYY.ZZZ",
    dry_run=True,  # Validate without modifying S3
)

# Regenerate Item metadata
generator.run()

# With explicit, non-standard bucket name (strict structure mode must be off)
generator_with_bucket = ItemGeneratorS3(
    "EO.XXX.YYY.ZZZ",
    bucket_name="custom-bucket",
    backup_items=True,
    strict_structure=False,
)
generator_with_bucket.run()

# Skip the ready_for_transfer safety check (see Safety Behavior below); enabled by default
generator_without_gate = ItemGeneratorS3(
    "EO.XXX.YYY.ZZZ",
    use_ready_for_transfer_flag=False,
)
generator_without_gate.run()

Strict structure mode

Strict structure mode is on by default. It insists that a delivery follows the standard:

1. Bucket name — the bucket must be named usergenerated-proposal-<collection id lowercased, "_" replaced by "-">. Dots are kept, because S3 bucket names cannot contain underscores. For example:

   Collection ID	Required bucket name
   EO.XXX.YYY.ZZZ	usergenerated-proposal-eo.xxx.yyy.zzz
   EO.ESA_X.YYY.ZZZ	usergenerated-proposal-eo.esa-x.yyy.zzz

   The bucket name is also derived this way when --bucket-name is omitted. In strict mode, an explicit --bucket-name or a custom --s3-bucket-prefix that gives any other name is rejected.

2. Bucket layout — the bucket root must hold a folder named exactly the collection ID (case-sensitive), with the delivery (data/ and metadata/) inside it:

   usergenerated-proposal-eo.xxx.yyy.zzz/
   └── EO.XXX.YYY.ZZZ/
       ├── data/
       └── metadata/
   

   The flat layout (data/ and metadata/ at the bucket root) is rejected. Other top-level content in the bucket is tolerated.

A breach raises InvalidStructureError (a ConfigError, exit code 1) that names the expected bucket or folder. The check runs before anything is mirrored or written. The bucket name is checked when ItemGeneratorS3 is constructed, and the layout at the start of run().

To accept any bucket name and either layout, switch strict mode off: use --no-strict-structure on the CLI, strict_structure=False on ItemGeneratorS3 or promote_collection, or S3_STRICT_STRUCTURE=false for the demo scripts. For promote_collection, only the source bucket is checked.

Safety Behavior

The regeneration workflow includes several safety measures:

1. Validates credentials and bucket structure before processing. In strict structure mode (default), also requires the standard bucket name and a <collection_id>/ folder at the bucket root.
2. Mirrors S3 content locally and ensures local processing under <collection_id>/.
3. By default (use_ready_for_transfer_flag=True), requires "ready_for_transfer": true in the mirrored collection's metadata/collection_config.json; aborts with an error if the field is missing or false (this check applies even with --dry-run). Skip this check via --no-use-ready-for-transfer-flag on the CLI, the use_ready_for_transfer_flag=False constructor argument, or USE_READY_FOR_TRANSFER_FLAG=false for the demo script.
4. Clears the mirrored copy of metadata/items, then regenerates Item metadata locally with existing generation logic. This is what makes regeneration a replacement rather than a merge: an item in S3 with no matching item folder under data/ is reported (... have no matching item folder under 'data/' ...) and not re-uploaded, instead of surviving every run. The S3 backup in step 5 is the recovery path if a folder was removed by mistake.
5. Backs up existing <collection_id>/metadata/items by default (can be disabled).
6. Replaces only <collection_id>/metadata/items in S3 (never modifies data/).
7. With --dry-run, performs local validation and regeneration, then prints planned S3 actions without modifying the bucket.

Mirroring Behavior

The mirroring process downloads real files selectively to improve performance:

1. Downloads real files only for metadata/* and item_config.json keys.
2. Writes per-folder manifest files (usergenerated_s3_folder_manifest.jsonl) for discovered objects.
3. Uses manifest entries during item generation so full non-metadata assets are not required locally.
4. Supports optional placeholder file creation for non-fetched objects (disabled in the S3 regeneration workflow).
5. Recreates folder-marker objects (keys ending in /) as empty local folders, with no manifest, so an empty Item folder in the bucket is caught by the structure check.
6. Logs a summary with discovered, downloaded, and failed object counts, plus periodic progress logs during large scans.

Maintainer Notes

Recent internal refactoring improved testability and packaging workflows.

1. The primary command is now generate-item-metadata <COLLECTION_ID>.
2. generate_item_metadata.py is retained as a compatibility wrapper.
3. Internal logic was modularized into reusable helpers under usergenerated/ for:

• collection path resolution
• item folder discovery by configured granularity
• item config precedence/loading
• simplified and hierarchy-based datetime resolution

4. This split enables unit-heavy testing of pure logic while keeping entry-point orchestration thin.

Optional S3 Upload

Automatic upload is configured at runtime via ItemGenerator constructor parameters or CLI flags (not via module constants). Upload is opt-in and off by default: without is_upload_s3=True / --upload-s3, no AWS credentials are read and nothing touches S3. When upload is requested, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY must be set — otherwise ItemGenerator fails at construction, before any item is generated.

Uploads follow the same strict structure mode as regenerate-item-metadata-s3 (on by default): the target bucket must be the standard usergenerated-proposal-<collection id lowercased, "_" replaced by "-"> (e.g. EO.ESA_X.YYY.ZZZ → usergenerated-proposal-eo.esa-x.yyy.zzz), and the collection is uploaded under a <collection id>/ folder. A --bucket-name / override_bucket_name or custom --s3-bucket-prefix that selects any other bucket fails with InvalidStructureError before any item is generated. Pass --no-strict-structure / strict_structure=False to upload to a non-standard bucket. The underscore replacement applies in both modes. Without --upload-s3, no bucket rule is checked.

Programmatic example:

from usergenerated import ItemGenerator

generator = ItemGenerator(
    "EO.XXX.YYY.ZZZ",
    is_upload_s3=True,
    s3_endpoint_url="https://s3.central.data.destination-earth.eu",
    s3_user_generated_bucket_prefix="usergenerated-proposal",
)
generator.run()

CLI example:

generate-item-metadata EO.XXX.YYY.ZZZ \
  --upload-s3 \
  --s3-endpoint-url "https://s3.central.data.destination-earth.eu" \
  --s3-bucket-prefix "usergenerated-proposal"

If upload is enabled, create a .env file at the project root with credentials:

AWS_ACCESS_KEY_ID="<your_access_key_id>"
AWS_SECRET_ACCESS_KEY="<your_secret_access_key>"

For manual upload workflows, see README-s3cmd.md.

S3 utility wrappers are available for programmatic execution around S3Tools:

from script_s3_create_bucket import run_create_bucket
from script_s3_move_bucket_contents import run_move_bucket_contents
from script_s3_upload_file_to_bucket import run_upload_file
from script_s3_upload_folder_to_bucket import run_upload_folder

run_create_bucket(
  aws_access_key_id="<key>",
  aws_secret_access_key="<secret>",
  s3_bucket_name="my-bucket",
)

run_upload_file(
  file_name="EO.XXX.YYY.ZZZ/metadata/collection.json",
  object_name="EO.XXX.YYY.ZZZ/metadata/collection.json",
  aws_access_key_id="<key>",
  aws_secret_access_key="<secret>",
  s3_bucket_name="my-bucket",
)

run_upload_folder(
  folder_path="EO.XXX.YYY.ZZZ/metadata/items",
  target_path="EO.XXX.YYY.ZZZ/metadata/items",
  aws_access_key_id="<key>",
  aws_secret_access_key="<secret>",
  s3_bucket_name="my-bucket",
)

run_move_bucket_contents(
  aws_access_key_id="<key>",
  aws_secret_access_key="<secret>",
  s3_bucket_name="my-bucket",
  s3_target_prefix="archive",
)

S3 Folder Upload Progress

When using run_upload_folder(...) or S3Tools.upload_folder_to_s3(...), uploads now display progress bars:

1. A global Total files bar showing overall folder upload progress.
2. Per-file bars (bytes transferred) for active uploads.

Current behavior:

1. Uploads run concurrently with a bounded worker pool.
2. Per-file progress bars are capped to a visible range and positions are reused for large uploads.
3. usergenerated_s3_folder_manifest.jsonl files and symlinks are skipped from upload discovery.
4. A final summary logs discovered, uploaded, and failed file counts.
5. Progress bars are drawn only when stderr is a terminal. Under a non-interactive runner (CI job, Kubernetes pod, cron entry) they are suppressed automatically, so captured logs contain the summary lines rather than thousands of bar refreshes. The same applies to the collection copier's Copying objects bar.

These wrappers accept explicit parameters first and only fall back to environment variables when a parameter is omitted.

Regenerate Item Metadata: Use the CLI command for routine operations (see Regenerate Item Metadata in S3):

regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --dry-run

The script_s3_regenerate_item_metadata.py file is kept as a small demonstrator of ItemGeneratorS3 programmatic usage. Use the CLI command or the ItemGeneratorS3 class directly for real workflows.

Copy a Collection Between Buckets

script_s3_copy_source_bucket_collection_to_target_bucket_collection.py copies a collection folder from a source S3 bucket/account to a target S3 bucket/account, using usergenerated.s3.s3_collection_copier.CollectionCopier. It:

• Detects the source bucket's BucketLayout (COLLECTION_ROOT or FLAT_ROOT) and normalizes the target to always be <collection_id>/... (COLLECTION_ROOT), fixing up FLAT_ROOT sources in the process.
• Never modifies the source bucket.
• Synchronises rather than merely appends: an object already in the target is skipped only when it is unchanged (same size, and the target copy is not older than the source); a changed object is overwritten, and an object the source no longer has is deleted from the target. After a run the target matches the source, which is exactly what the verifier below checks.
• Is safely re-runnable: an interrupted transfer can just be re-run to continue where it left off, and a repeated run after the source has moved on converges instead of drifting.
• Streams each object directly from source to target (no local disk buffering), since source and target buckets are assumed to use different credentials/accounts.

Configure both buckets in .env (the source variables are the same ones used elsewhere in this project; target variables are TARGET_-prefixed):

# Source bucket (existing variables, unchanged)
S3_ENDPOINT_URL="https://s3.central.data.destination-earth.eu"
S3_BUCKET_NAME="my-source-bucket"
AWS_ACCESS_KEY_ID="<source_access_key_id>"
AWS_SECRET_ACCESS_KEY="<source_secret_access_key>"

# Target bucket
TARGET_S3_ENDPOINT_URL="https://s3.central.data.destination-earth.eu"
TARGET_S3_BUCKET_NAME="my-target-bucket"
TARGET_AWS_ACCESS_KEY_ID="<target_access_key_id>"
TARGET_AWS_SECRET_ACCESS_KEY="<target_secret_access_key>"

S3_COLLECTION_ID="EO.XXX.YYY.ZZZ"
S3_FRESH_START="false"    # set true to delete the target's existing '<collection_id>/' prefix first
S3_DRY_RUN="false"        # set true to preview the copy plan without modifying the target
S3_DELETE_ORPHANS="true"  # set false for an additive-only copy that never removes target objects

Then run:

python script_s3_copy_source_bucket_collection_to_target_bucket_collection.py

S3_FRESH_START is the only way to trigger the destructive "clean the target path first" step (e.g. for a first run against a target that already has stray/partial contents); it is never triggered automatically. Omit it (or leave it false) for the default incremental, resume-friendly sync.

Two safety rules apply whenever S3_FRESH_START is enabled:

• If the source listing turns up no objects for the collection (wrong bucket, wrong collection id, a misdetected layout), the run aborts with S3OperationError and deletes nothing — the target copy is never replaced with an empty one. This check also fires under S3_DRY_RUN.
• If the delete only partially succeeds, the run fails with S3OperationError rather than copying on top of a half-cleared prefix and reporting success.

Under S3_DRY_RUN the plan reports the full source object count as "to copy" (fresh_start clears the target, so nothing is skipped) and logs how many objects the delete would remove.

S3_DELETE_ORPHANS (default true) controls the incremental path's own delete: target objects under <collection_id>/ that the source no longer has. Set it to false for an additive-only copy — but note the verifier will then fail with unexpected in target after any source-side deletion, because the two would no longer agree on what the collection contains. The same empty-source-listing guard applies: if the source lists no objects, the run aborts with S3OperationError rather than deleting the whole target copy as "orphaned" (this fires under S3_DRY_RUN too), and a partial delete fails the run before any object is copied.

Every run logs its plan in four categories — to copy / to overwrite (changed in source) / to delete (absent from source) / unchanged and skipped — so the log says why work happened.

Upgrading from 2.0.2 or earlier. Previously the copier compared keys only, so it never re-copied a changed object and never removed a deleted one. A second copy of a collection whose metadata/collection_config.json had been written back to (the ready_for_transfer handshake) therefore failed verification permanently with a size mismatch, and a same-size content change left the target silently stale. Both are fixed. The behavioural change to be aware of is that fresh_start=False now deletes target-only objects unless you pass delete_orphans=False.

Folder uploads use object keys relative to the selected local folder. For example, uploading ./EO.XXX.YYY.ZZZ/metadata/items with no target_path uploads files like a.json and nested/b.json at the bucket root. To place files under a collection or metadata prefix in the bucket, pass that prefix explicitly as target_path.

Verify a Collection Copy

script_s3_verify_collection_copy.py is a read-only smoke test to run right after the copy script above completes: it uses usergenerated.s3.s3_collection_copy_verifier.CollectionCopyVerifier to list the collection's objects in both buckets and confirms the target has exactly the same relative keys as the source, with matching sizes for every key. Neither bucket is modified.

It reuses the same .env variables as the copy script above (no new variables needed):

python script_s3_verify_collection_copy.py

The script exits 0 and prints Collection copy verification passed: True when the target matches the source. On a mismatch CollectionCopyVerifier.run() raises S3OperationError whose message summarises the missing, unexpected, and/or size-mismatched keys (sampled, up to 20 per category); the script reports that one line and exits 1.

Similarly, CollectionCopier.run() (the copy script above) raises S3OperationError — naming the failed-object count and a sample of failed keys — if any object fails to copy. Objects that did copy are left in place, so re-running resumes from where it stopped.

S3 helper code lives under usergenerated/s3/.

Promote a Collection (regenerate → copy → verify → read, in one call)

usergenerated.s3.collection_promoter.promote_collection (also from usergenerated import promote_collection) runs the whole "a collection has been staged in a source bucket; validate it, copy it to a target bucket, and hand me its items" workflow as a single call, in strict order:

1. ItemGeneratorS3(...).run() — regenerate STAC item metadata in the source bucket (source credentials; the ready_for_transfer safety gate stays on; data/ is untouched).
2. CollectionCopier(...).run() — copy the collection source bucket → target bucket.
3. CollectionCopyVerifier(...).run() — confirm the target matches the source key-for-key.
4. Build a PromotedCollection read accessor over the target copy and call on_ready(handle) exactly once.

on_ready is a callback you provide. Its argument exposes just two methods:

• handle.read_collection() -> dict — the target copy's collection.json (wraps get_collection_from_s3).
• handle.iter_items() -> Iterator[dict] — the target copy's STAC Items, streamed one at a time (wraps iter_collection_items_from_s3; lazy — consume it inside the callback).

Any domain-specific downstream work (database load, re-publish, indexing) stays entirely in your on_ready; promote_collection owns no datastore and never returns a status — a normal return means every stage passed and on_ready ran.

• on_ready runs only after copy and verify have both succeeded. Any earlier failure raises (S3OperationError / ConfigError / CredentialsError / FileNotFoundError, and FileExistsError from regeneration) and on_ready is never called.
• dry_run=True runs regeneration/validation only (via ItemGeneratorS3(dry_run=True)); the copier and verifier are never constructed and on_ready is never called.
• fresh_start=True passes through to CollectionCopier, deleting the target <collection_id>/ prefix before copying. The source bucket is never modified. It aborts without deleting if the source listing is empty, and fails the run if the delete only partly succeeds.
• delete_orphans=True (the default) passes through to CollectionCopier: the copy removes target objects the source no longer has, so a repeat promotion after items were dropped or rewritten still verifies. Pass delete_orphans=False for an additive-only copy.
• asset_href_prefix="..." (optional) is used for the regenerated item asset hrefs, overriding any asset_href_prefix in the source collection_config.json or an item_config.json; "" forces no prefix. The source config is not modified. See Overriding the prefix at run time.
• Before any stage runs (dry run included), promote_collection checks that both buckets exist for the given credentials (S3OperationError otherwise) and that no endpoint or credential is blank (CredentialsError).
• strict_structure=True (the default): the source bucket must be the standard usergenerated-proposal-<id> bucket and the target the standard usergenerated-<id> bucket, and the source must hold the delivery under <collection_id>/, or the call raises InvalidStructureError before anything runs. Pass strict_structure=False for non-standard buckets. See Strict structure mode.
• Idempotent / resumable: re-running after a failure repeats the skip-aware copy, re-verifies, and re-invokes on_ready. Side effects are only on the source (item regen, plus a timestamped items backup when backup_items=True) and the target (the copy).

from usergenerated import PromotedCollection, promote_collection


def on_ready(collection: PromotedCollection) -> None:
    stac = collection.read_collection()
    item_count = sum(1 for _ in collection.iter_items())
    print(f"{stac['id']}: {item_count} item(s) ready in the target bucket")


promote_collection(
    collection_id="EO.XXX.YYY.ZZZ",
    source_endpoint_url="https://s3.central.data.destination-earth.eu",
    source_bucket_name="usergenerated-proposal-eo.xxx.yyy.zzz",  # standard name (strict structure mode)
    source_aws_access_key_id="<source_access_key_id>",
    source_aws_secret_access_key="<source_secret_access_key>",
    target_endpoint_url="https://s3.central.data.destination-earth.eu",
    target_bucket_name="my-target-bucket",
    target_aws_access_key_id="<target_access_key_id>",
    target_aws_secret_access_key="<target_secret_access_key>",
    on_ready=on_ready,
)

script_s3_promote_collection.py is a runnable .env-driven demo whose on_ready logs the promoted collection's id and item count. It reuses the same variables as the copy and regenerate scripts (S3_COLLECTION_ID, S3_ENDPOINT_URL, S3_BUCKET_NAME, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, TARGET_S3_ENDPOINT_URL, TARGET_S3_BUCKET_NAME, TARGET_AWS_ACCESS_KEY_ID, TARGET_AWS_SECRET_ACCESS_KEY, S3_DRY_RUN, S3_FRESH_START), plus optional keys S3_LOCAL_ROOT (default tmp_structure_mirror), S3_BACKUP_ITEMS (default true), S3_ASSET_HREF_PREFIX (unset = use the configured prefix; set to empty = force no prefix) and S3_STRICT_STRUCTURE (default true).

python script_s3_promote_collection.py

Manipulate Collection Config in S3

script_s3_collection_config_manipulation.py demonstrates the helpers in usergenerated.s3.s3_collection_config_tools for reading and updating a collection's collection_config.json directly in S3, without a local mirror. Unlike the verify script above, it mutates the live bucket. For a given collection it:

1. Resolves the config key with resolve_collection_config_key and prints the full collection_config.json.
2. Reads the ready_for_transfer flag via get_ready_for_transfer.
3. Sets ready_for_transfer to false via set_ready_for_transfer.
4. Adds an ad-hoc last_transferred field (current UTC time, ISO 8601) via update_collection_config.

Setting ready_for_transfer to false engages the regenerate-item-metadata-s3 safety gate described in Safety Behavior — a subsequent regeneration run will refuse to proceed until the flag is set back to true (or the check is explicitly skipped).

For programmatic use the same module exposes two reusable building blocks so a caller does not have to chain layout detection and a hand-written read-modify-write:

• resolve_collection_config_key(s3_client, bucket_name, collection_id) — locate collection_config.json in the bucket without handling BucketLayout yourself.
• update_collection_config(s3_client, bucket_name, key, changes) — one read-modify-write for an arbitrary set of fields. changes is either a mapping merged into the config (use the exported REMOVE sentinel as a value to delete a key) or a callback that mutates the dict in place. Keys you do not touch keep their value, order, and the file's indent=4 formatting; it raises ConfigError if the object is missing or not a JSON object and S3OperationError on an S3 read/write failure. set_ready_for_transfer is a one-line wrapper over it.

Configure the target bucket in .env (the same single-bucket variables used by the regeneration demo script):

S3_ENDPOINT_URL="https://s3.central.data.destination-earth.eu"
S3_BUCKET_NAME="my-bucket"
AWS_ACCESS_KEY_ID="<access_key_id>"
AWS_SECRET_ACCESS_KEY="<secret_access_key>"
S3_COLLECTION_ID="EO.XXX.YYY.ZZZ"

Then run:

python script_s3_collection_config_manipulation.py

Report Bucket Information

script_s3_bucket_information.py demonstrates usergenerated.s3.s3_bucket_info.get_bucket_information, which makes a single paginated sweep of a bucket (or a prefix within it) and prints a JSON report: file count, total and average size, the 5 largest files, oldest/most-recent modification times, a count+size breakdown by top-level folder, a generated_at timestamp, and a sample file listing capped at 50 entries (with a sample_truncated flag) so you never get thousands of rows back. Every size in the report (total_size_bytes, average_file_size_bytes, and each size_bytes in largest_files, top_level_prefixes, and sample_files) has a ..._human/size_human counterpart, e.g. "total_size_human": "94.2 MB". It is read-only — the bucket is never modified.

Writing the report to output/<YYYYMMDDTHHMMSS>_Bucket_Information.json (the timestamp matches generated_at in the JSON) is off by default — the report is only printed to stdout. Turn it on with either S3_WRITE_OUTPUT_FILE="true" in .env, or write_output_file=True when calling run_get_bucket_information programmatically; the output/ directory is created if needed.

Configure the bucket in .env (the same single-bucket variables used by the config manipulation script above); S3_PREFIX, S3_BUCKET_INFO_SAMPLE_LIMIT, S3_WRITE_OUTPUT_FILE, and S3_BUCKET_INFO_OUTPUT_DIR are optional:

S3_ENDPOINT_URL="https://s3.central.data.destination-earth.eu"
S3_BUCKET_NAME="my-bucket"
AWS_ACCESS_KEY_ID="<access_key_id>"
AWS_SECRET_ACCESS_KEY="<secret_access_key>"
S3_PREFIX=""                        # optional: scope the report to one folder/collection
S3_BUCKET_INFO_SAMPLE_LIMIT="50"    # optional: cap on the sample file listing
S3_WRITE_OUTPUT_FILE="false"        # optional: set true/1/yes/on to also write the JSON report to disk
S3_BUCKET_INFO_OUTPUT_DIR="output"  # optional: where the timestamped JSON report is written

Then run:

python script_s3_bucket_information.py

Copy Collection Metadata Locally

script_s3_get_metadata.py demonstrates usergenerated.s3.s3_collection_metadata_downloader.download_collection_metadata, which copies a collection's metadata/ folder from S3 to a local destination. The source bucket's layout (whether it holds <collection_id>/metadata/ or a top-level metadata/) is detected automatically, so the caller never needs to know or check it beforehand — the result always lands under a folder named for the collection ID, e.g. <destination_root>/EO.XXX.YYY.ZZZ/metadata. It is read-only with respect to S3 — nothing in the source bucket is modified.

The destination root is configurable via S3_METADATA_DESTINATION_ROOT (or the destination_root argument when calling run_get_metadata programmatically); when unset, it defaults to a fresh timestamped folder under the OS temp directory (/tmp on Linux, %TEMP% on Windows), e.g. <os-temp-dir>/usergenerated_tmp/20260824T101500/EO.XXX.YYY.ZZZ/metadata.

Objects are downloaded concurrently — one worker thread per object, bounded by the max_workers argument (default MAX_METADATA_DOWNLOAD_WORKERS = 16, capped at 32). The single s3_client is shared across those threads, so build it with a matching connection pool: create_s3_client(..., max_pool_connections=MAX_METADATA_DOWNLOAD_WORKERS) (botocore defaults to 10). If any object fails to download the whole call raises S3OperationError (a RuntimeError subclass) rather than returning a partial folder; once a systemic problem has failed ~20 objects it stops submitting new downloads instead of grinding through every remaining key.

For callers that only need the files for the duration of a block — parse collection.json, stream items/ somewhere, done — use the downloaded_collection_metadata context manager instead of calling download_collection_metadata directly; on exit it removes the <collection_id>/ subtree it downloaded, and if it also generated the timestamped destination root (no explicit destination_root given) it removes that too, so nothing is left under /tmp. An explicit destination_root you pass in is otherwise left in place:

with downloaded_collection_metadata(s3_client, bucket, collection_id) as metadata_folder:
    collection = get_stac_collection(metadata_folder)
    for item in iter_items(metadata_folder):
        ...
# the timestamped <os-temp-dir>/usergenerated_tmp/<...> directory is gone here

The same module also provides small supporting functions for reading the downloaded metadata: get_stac_collection/get_collection_config load collection.json/collection_config.json as dicts, and iter_items/iter_item_batches stream STAC Items out of the downloaded items/ folder — one item at a time, or in fixed-size batches — rather than loading every item into memory at once. This matters for collections with very large item counts (hundreds of thousands), where materializing every item as a single list would be memory-intensive. The batch size defaults to METADATA_DOWNLOADER_DEFAULT_ITEM_BATCH_SIZE (500) and is configurable via S3_METADATA_ITEM_BATCH_SIZE (or the item_batch_size argument to run_get_metadata).

If you only need to walk the items once and don't need them to persist, iter_collection_items_from_s3 / iter_collection_item_batches_from_s3 (and get_collection_from_s3 for collection.json) stream STAC Items straight from the bucket with no local copy written and no cleanup to do — a drop-in for iter_items / iter_item_batches / get_stac_collection. The bucket layout is detected automatically, item bodies are fetched with a bounded concurrent prefetch (max_workers defaults to MAX_METADATA_DOWNLOAD_WORKERS, capped at 32), and any fetch or parse failure raises S3OperationError rather than yielding a partial result. Keep download_collection_metadata for when the files must stay on disk or be read more than once. Setting S3_METADATA_STREAM_ITEMS to a truthy value (or passing stream_items=True to run_get_metadata) runs script_s3_get_metadata.py on this no-download path.

Configure the collection and bucket in .env (the same single-bucket variables used by the config manipulation script above); S3_METADATA_DESTINATION_ROOT, S3_METADATA_ITEM_BATCH_SIZE, and S3_METADATA_STREAM_ITEMS are optional:

S3_COLLECTION_ID="EO.XXX.YYY.ZZZ"
S3_ENDPOINT_URL="https://s3.central.data.destination-earth.eu"
S3_BUCKET_NAME="my-bucket"
AWS_ACCESS_KEY_ID="<access_key_id>"
AWS_SECRET_ACCESS_KEY="<secret_access_key>"
S3_METADATA_DESTINATION_ROOT=""  # optional: local folder to copy metadata/ into (default: <os-temp-dir>/usergenerated_tmp/<timestamp>)
S3_METADATA_ITEM_BATCH_SIZE=""  # optional: items per batch when streaming items/ (default: 500)
S3_METADATA_STREAM_ITEMS=""  # optional: truthy = skip the local download, stream items straight from S3

Then run:

python script_s3_get_metadata.py

Inspect a Deployed Collection

A collection is deployed once it is served by the DestinE Data Lake HDA STAC API at https://hda.data.destination-earth.eu/stac/v2/collections. This is the check that comes after a collection has been promoted to the target bucket and picked up by DEDL.

Some collections are protected by a role and are simply absent for anonymous callers, so script_deployed_information.py always makes both attempts and reports them side by side:

• anonymous — a plain request, no credentials.
• authenticated — a request carrying a DEDL access token as an Authorization: Bearer header. The token comes from a DESP username/password via the destinelab token exchange, and makes the collections your account is entitled to see visible.

From the two outcomes it derives:

• deployed — either attempt found the collection.
• requires_authentication — only the authenticated attempt found it, i.e. the collection is deployed but protected.

The script is read-only: it issues HTTP GETs only, and writes nothing to the STAC API or to S3. The DESP password and the DEDL token never appear in the report or in the logs.

Install the optional dependency

The authenticated attempt needs destinelab, which ships as an optional extra so anonymous lookups work on a plain install:

uv sync --extra deployed

Without it (or without DESP credentials), the authenticated attempt is reported as skipped and the run still succeeds with the anonymous result.

Configuration

Add the keys you need to .env:

DEPLOYED_COLLECTION_ID="EO.XXX.YYY.ZZZ"
DESP_USERNAME="<desp_username>"  # optional: omit to run the anonymous lookup only
DESP_PASSWORD="<desp_password>"  # optional: omit to run the anonymous lookup only
DEPLOYED_STAC_BASE_URL=""  # optional: STAC API base URL (default: https://hda.data.destination-earth.eu/stac/v2)
DEPLOYED_AUTHENTICATE=""  # optional: falsy = skip the authenticated attempt entirely (default: true)
DEPLOYED_OUTPUT_DIR=""  # optional: folder for the output file (default: output)
DEPLOYED_WRITE_OUTPUT_FILE=""  # optional: falsy = skip writing the collection file (default: written)

The output file contains only the STAC Collection — not the surrounding report — so it can be diffed straight against a local metadata/collection.json. Before writing, the document is loaded into PySTAC and validated:

• Normalized — the file is PySTAC's own serialization, so its field ordering is standardized rather than whatever the server happened to return, and successive runs produce diffable files. Extension and vendor fields (dedl:*, cube:*, sci:*, …) are preserved.
• Validated — the result is reported in the console report as stac_valid and stac_validation_error, and the report also gains the output_file path. Validation is reported, not enforced: a deployed collection that fails validation is still written, since it is not yours to fix and is exactly the file worth inspecting. A document PySTAC cannot read as a Collection at all raises CollectionValidationError.

The filename carries a filename-safe form of the collection id (every run of non-alphanumeric characters collapses to a single underscore, so EO.XXX.YYY.ZZZ becomes EO_XXX_YYY_ZZZ), so several collections can share one output folder:

output/20260904T164512_EO_XXX_YYY_ZZZ_Deployed_Collection.json

When both attempts found the collection the authenticated body is written (it is the only one that can see a protected collection; for a public one the two are identical). Nothing is written when neither attempt found it.

Then run:

python script_deployed_information.py

Example output for a deployed but role-protected collection (the full STAC Collection body is included under each successful attempt, truncated here):

{
    "collection_id": "EO.XXX.YYY.ZZZ",
    "base_url": "https://hda.data.destination-earth.eu/stac/v2",
    "anonymous": {
        "attempted": true,
        "found": false,
        "status_code": 404,
        "error": "Collection 'EO.XXX.YYY.ZZZ' is not visible to this caller (HTTP 404).",
        "collection": null
    },
    "authenticated": {
        "attempted": true,
        "desp_username": "user@example.org",
        "roles": ["..."],
        "found": true,
        "status_code": 200,
        "error": null,
        "collection": { "type": "Collection", "id": "EO.XXX.YYY.ZZZ", "...": "..." }
    },
    "deployed": true,
    "requires_authentication": true,
    "generated_at": "2026-09-04T16:23:14.859202+00:00"
}

Programmatic Usage

The helpers behind the script live in usergenerated.deployed and can be used directly:

from usergenerated.deployed.auth import authenticate_dedl
from usergenerated.deployed.stac_api import get_collection, list_collection_ids

# Anonymous - only public collections are visible.
lookup = get_collection("EO.ESA.DAT.SENTINEL-2.MSI.L1C")
print(lookup.found, lookup.status_code)

# Authenticated - protected collections the account is entitled to see become visible.
authentication = authenticate_dedl("<desp_username>", "<desp_password>")
protected = get_collection("EO.XXX.YYY.ZZZ", token=authentication.token)
print(authentication.roles, protected.collection)

# Every collection id visible to this caller.
print(len(list_collection_ids(token=authentication.token)))

get_collection returns a CollectionLookup (found, status_code, collection, error). A collection that is not visible is reported as found=False with the HTTP status - not an exception. Transport failures, unexpected statuses, and unparseable bodies raise DeployedApiError; missing DESP credentials or an unusable token raise CredentialsError; a missing destinelab install raises ConfigError telling you to run uv sync --extra deployed.

Compare a Deployed Collection with the Local collection.json

script_compare_collections.py checks that two STAC Collection files hold the same content — typically the deployed collection written above against the metadata/collection.json it was published from:

python script_compare_collections.py \
    output/20260914T142622_EO_FMI_DAT_DESTINE_CLIMATE_WILDFIRE_FWI_Deployed_Collection.json \
    EO.FMI.DAT.DESTINE_CLIMATE_WILDFIRE_FWI/metadata/collection.json

• Key order is irrelevant. Both files are loaded through pystac first, so an unsorted, hand-written collection.json compares on equal terms with the pystac-sorted deployed file. List order still counts (bbox, keywords, providers).
• Server-managed fields are ignored by default: links, created, updated, published, federation and summaries/federation:backends. A container left empty by that (e.g. summaries) counts as absent.
• stac_extensions is always compared. Every extension the deployed collection declares must also be declared in collection.json — including the timestamps extension (https://stac-extensions.github.io/timestamps/v1.1.0/schema.json), which the deployed collection carries because of its created/updated/published fields.
• --ignore FIELD adds a field to ignore, as a /-separated path from the document root (e.g. --ignore assets/thumbnail); --ignore-extension URI_PREFIX adds a stac_extensions prefix; --no-default-ignores compares everything. Both flags are repeatable.

It prints a JSON report listing every difference with its path (e.g. extent/spatial/bbox[0][2], providers[1]) and kind (value_mismatch, only_in_reference, only_in_candidate), then exits 0 when the collections match and 1 when they differ, so it can gate a pipeline. It reads the two files only; nothing is written and no network request is made.

Programmatically:

from pathlib import Path
from usergenerated.deployed.collection_comparator import compare_collection_files

comparison = compare_collection_files(Path("deployed.json"), Path("collection.json"))
for difference in comparison.differences:
    print(difference.path, difference.kind.value, difference.reference, difference.candidate)

Runtime Initialization

Logging initialization is now explicit and idempotent.

1. Entry-point scripts call setup_logging() from usergenerated.logging_config at startup.
2. Environment loading (load_dotenv) is also called explicitly at startup.
3. Importing modules should not configure logging implicitly.

This keeps behavior stable regardless of import order and avoids duplicate log handlers.

Windows

The package is tested on Linux and Windows (CI runs the suite on both). A few platform notes:

• Temp directory. Features that mirror or download from S3 (regenerate-item-metadata-s3, script_s3_get_metadata.py) use the OS temp directory (%TEMP% on Windows) for their working copy, not a hardcoded /tmp. Override it with S3_METADATA_DESTINATION_ROOT where supported.

• Content-based MIME sniffing. itemhelper.guess_mime_type_advanced() needs the optional magic extra (uv sync --extra magic / pip install "usergenerated[magic]") and a native libmagic, which is not present on a default Windows install. Nothing in the standard workflow calls it — media types are inferred from the file extension.

• Symlinks in data/. Creating symlinks on Windows needs SeCreateSymbolicLinkPrivilege (Developer Mode or an elevated shell). The tools skip symlinks on upload regardless.

• S3 keys with unusual names. Keys containing characters that are illegal in Windows filenames (: * ? " < > |), or that differ only in case, are sanitised to a safe local name when mirrored; the original key is preserved in the per-folder manifest.

• Long paths (the 260-character limit). Windows refuses to create a path longer than 260 characters unless long-path support is switched on. This is reachable in practice: an item folder name embeds the whole collection id (<collection_id>_<start>_<end>), so a deep working directory plus a long collection id can cross the limit while mirroring. The tools detect this before creating anything and fail with a message naming the path and its length, rather than the misleading WinError 3: The system cannot find the path specified. Fix it by running from a shallower directory, passing a shorter --local-root / S3_METADATA_DESTINATION_ROOT, or enabling long paths system-wide:

  # Run as Administrator, then reboot.
  Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem' `
      -Name LongPathsEnabled -Value 1
  

• Collection folder name case. The Windows filesystem is case-insensitive, so a folder named eo.xxx.yyy.zzz would otherwise satisfy a request for EO.XXX.YYY.ZZZ and then fail for a colleague on Linux. Collection ids are case-sensitive, so this is reported as an error on every OS — rename the folder to match the id exactly.

• Console output. Log records and progress bars are written as UTF-8 with a safe fallback, so a non-ASCII filename or collection description does not break a legacy cmd.exe console.

Exit codes and failure output

Every console command (generate-item-metadata, regenerate-item-metadata-s3) and root script_s3_*.py shares one convention:

Exit code	Meaning
0	success
1	an expected failure (bad config, missing credentials, a missing file or folder you named, invalid collection.json, a failed S3 operation) — reported as a single ERROR log line, no traceback; or an unexpected internal error, which keeps its traceback
2	command-line usage error (bad or missing arguments)

Troubleshooting

1. Generation fails with folder-level errors: Set item_folder_level to match your actual data/ layout.

2. Generation fails with "non-conforming path(s) under 'data/'": The structure conformance check rejected your layout. Each reported line names the offending path and the remedy - see Structure Conformance Check.

3. Missing item_config.json errors: Set item_config_optional to true if item config files are intentionally omitted.

4. Collection id validation errors: Ensure root folder name and collection_config.json > id represent the same collection id.

5. Upload credential errors: Ensure .env contains valid AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY when upload is enabled.

External References

1. DestinE Data Lake - Promote User Data to become DestinE data
2. STAC Collection specification
3. STAC Item specification