User Generated Data Process

Table of Contents:

1. Overview
2. User Workflow
3. Required Structure
4. Metadata Files
   • collection.json
   • collection_config.json
5. Generate Item Metadata
6. Regenerate Item Metadata in S3
7. Maintainer Notes
8. Optional S3 Upload
9. Runtime Initialization
10. Troubleshooting
11. External References

This project helps you prepare a User Generated collection for DestinE Data Lake review.

Use this guide to:

1. Structure your collection folders and data assets.
2. Configure metadata files.
3. Generate STAC Item metadata.

For contributor and maintenance documentation, see docs/development.md.

User Workflow

1. Create your collection root folder using the provided collection id, for example EO.XXX.YYY.ZZZ.
2. If needed, place the collection in a subfolder under your current working directory and pass that subfolder with --collection-path.
3. Add metadata files under metadata/.
4. Organize data assets under data/ according to your configured folder level.
5. Run generate_item_metadata.py to validate and generate STAC Items under metadata/items/.

If you mirror a collection from S3 before regenerating items, mirrored data folders may contain usergenerated_s3_folder_manifest.jsonl files instead of placeholder dummy assets. Item generation detects these manifests automatically and uses their entries as the authoritative asset list.

Reference collection structure is available in EO.XXX.YYY.ZZZ.

Required Structure

Expected top-level structure:

EO.XXX.YYY.ZZZ/
  data/
  metadata/
    collection.json
    collection_config.json
    items/

Notes:

1. Collection ids are case-sensitive and typically uppercase with . separators.
2. The metadata/items/ folder is where generated Item JSON files are written.

Metadata Files

collection.json

How to Complete collection.json

Use metadata/collection.json as the authoritative STAC Collection metadata file for your dataset.

---

Required STAC Collection fields

The following fields must be completed and valid:

1. type — Must be set to Collection.
2. stac_version — Use the current version (1.1.0). generate-item-metadata rejects the collection if stac_version is missing or set to anything else.
3. id — Your collection identifier. This must match:
   • your root folder name
   • collection_config.json > id
4. title — A human-readable name for your collection.
5. description — A detailed description of the dataset.
6. dedl:short_description — A concise version of the description. Displayed in the DestinE DataLake UI.
7. license — The license under which the data is distributed.
8. extent — Must include both spatial and temporal components.
9. links — Must be present as an array.
10. assets — Include if you have collection-level assets; otherwise use {}.

---

Recommended best practices

• Keep the id consistent across:
  • the collection root folder name
  • collection_config.json > id
• Use a clear title and a meaningful description that explains:
  • dataset content
  • data source(s)
  • intended use cases
• Ensure extent is accurate so item-level metadata is interpreted correctly.
• Only update necessary links (typically describedby and cite-as).
• Keep type and stac_version unchanged unless upgrading STAC versions.

---

Field-specific guidance

links

• Always include at least:
  • license → link to license terms
  • cite-as → DOI or citation reference
  • describedby → external dataset documentation
• Ensure links are stable and publicly accessible.
• Use meaningful title values so users understand what each link represents.

---

extent

• Spatial (bbox):

  • Format: [west, south, east, north]
  • Use WGS84 coordinates (EPSG:4326).
  • Ensure the bounding box fully covers your dataset.

• Temporal (interval):

  • Use ISO 8601 format (YYYY-MM-DDTHH:MM:SSZ).
  • Use:
    • a closed interval [start, end] for fixed datasets
    • [start, null] for ongoing datasets

---

license

• Prefer SPDX identifiers (e.g. CC-BY-4.0, MIT, Apache-2.0).
• Use:
  • proprietary for restricted datasets
  • various if multiple licenses apply
• If using proprietary or various:
  • explain clearly in description
  • provide license links in links

---

providers

• List all organizations involved in the dataset lifecycle.
• Common roles:
  • producer → created the data
  • processor → processed or transformed the data
  • licensor → owns or distributes the data rights
  • host → (optional) hosting platform
• Provide:
  • name (required)
  • url (recommended)
• Include multiple providers if applicable rather than combining roles into one.

---

sci:publications

• Use this field to reference scientific publications or datasets.
• Include:
  • sci:doi → DOI identifier
  • sci:citation → human-readable citation
• Add multiple entries if there are several related publications.
• Ensure DOIs resolve correctly (test the links).

---

keywords

• Provide relevant, searchable terms describing the dataset.
• Use:
  • domain-specific terminology (e.g. Land Cover, Atmospheric Data)
  • common synonyms where helpful
• Avoid overly generic terms (e.g. “data”, “analysis”).

---

assets

• Use for collection-level resources (not item-level data).
• Common examples:
  • thumbnail → preview image
  • overview → documentation PDF
• Include:
  • href (URL)
  • type (MIME type, e.g. image/png)
  • roles (e.g. thumbnail, overview)
• If no assets are needed, use an empty object:

  "assets": {}
  

collection_config.json

This controls how Item metadata generation behaves. Base example in EO.XXX.YYY.ZZZ/metadata/collection_config.json.

Configuration Options

The following options are user-facing and supported during item generation:

Key	Type	Default	Description
id	string	required	Collection id. Must match your collection root folder id.
item_folder_level	string	DD	Where item folders are discovered. Allowed values: YYYY, MM, DD, NONE.
item_config_optional	boolean	false	If false, each item folder is expected to contain item_config.json.
item_asset_ignore_list	list[string]	[]	File names to ignore when creating STAC assets.
media_type_overrides	object	{}	Optional extension to MIME type map used before default detection, e.g. { ".grib2": "application/x-grib2" }.
role_overrides	object	{}	Optional MIME type to role map. Overrides default data/metadata classification. Allowed roles: data, metadata, thumbnail, overview. E.g. { "application/x-grib2": "data" }. File-name regex patterns (thumbnail_regex, overview_regex) still take priority.
thumbnail_regex	string	unset	Regex used to detect thumbnail assets and assign thumbnail role.
overview_regex	string	unset	Regex used to detect overview assets and assign overview role.
additional_property_keys	list[string]	[]	Additional property suffix keys parsed from item folder names (standard mode).
item_folder_naming_convention_type	string	standard	Naming convention mode. Allowed values: standard, non_standard_1.
properties	object	{}	Additional properties merged into each generated Item.
bbox	list[number]	unset	Bounding box [minx, miny, maxx, maxy] applied to generated items.
item_date_overide	string	unset	Simplified mode override date in YYYYMMDD format.
asset_href_prefix	string	unset	Optional prefix prepended to every generated asset href, e.g. a public bucket URL such as https://s3.central.data.destination-earth.eu/. A trailing slash is normalized; an empty string behaves as unset.

Important: the accepted key name is item_date_overide (spelling as shown).

Folder Granularity Options

Set item_folder_level in collection_config.json.

1. DD: item folders are expected under data/YYYY/MM/DD/<item_folder>.
2. MM: item folders are expected under data/YYYY/MM/<item_folder>.
3. YYYY: item folders are expected under data/YYYY/<item_folder>.
4. NONE: simplified mode. Any folder directly under data/ becomes an Item.

Standard Mode Example

{
  "id": "EO.XXX.YYY.ZZZ",
  "item_asset_ignore_list": ["item_config.json"],
  "media_type_overrides": {
    ".grib2": "application/x-grib2"
  },
  "item_config_optional": false,
  "item_folder_level": "DD",
  "thumbnail_regex": "^thumbnail",
  "overview_regex": "^overview",
  "additional_property_keys": []
}

Simplified Mode Example

Simplified mode should only be used with agreement from support.

{
  "id": "EO.XXX.YYY.ZZZ",
  "item_asset_ignore_list": ["item_config.json"],
  "item_folder_level": "NONE",
  "item_date_overide": "20240101",
  "thumbnail_regex": "^thumbnail",
  "overview_regex": "^overview",
  "additional_property_keys": [],
  "bbox": [-10.0, 35.0, 10.0, 60.0]
}

Simplified mode datetime behavior:

1. If item_date_overide is set, that date is used for all generated Items.
2. Otherwise, if the first folder level under data/ is a year, YYYY-01-01 is used.
3. Otherwise, the generation date is used.

Asset Href Prefix

By default, generated asset hrefs are relative to the collection root, e.g.:

"href": "data/2024/11/15/EO.XXX.YYY.ZZZ_20241115T000000_20241115T235959/20241115.png"

Set asset_href_prefix in collection_config.json to prepend a fixed prefix (typically a public bucket URL) to every generated asset href:

{
  "id": "EO.XXX.YYY.ZZZ",
  "asset_href_prefix": "https://s3.central.data.destination-earth.eu/"
}

which produces:

"href": "https://s3.central.data.destination-earth.eu/data/2024/11/15/EO.XXX.YYY.ZZZ_20241115T000000_20241115T235959/20241115.png"

The prefix applies to every generated asset regardless of role (data, metadata, thumbnail, overview); item/collection self links are unaffected. This works identically for generate-item-metadata, regenerate-item-metadata-s3, and promote_collection — whichever collection_config.json is in effect for the collection being processed is the one consulted.

Per-Item Configuration

Each item folder can include an item_config.json file. Typical per-item overrides:

1. bbox
2. properties
3. Asset-related regex fields when needed

When present, item-level values override collection-level values.

Generate Item Metadata

CLI Usage

Primary command (after installing the package):

generate-item-metadata EO.XXX.YYY.ZZZ

This default expects the collection at ./EO.XXX.YYY.ZZZ.

Optional collection subfolder path:

generate-item-metadata EO.XXX.YYY.ZZZ --collection-path toto

With this option, the collection is resolved at ./toto/EO.XXX.YYY.ZZZ. For safety, --collection-path is currently restricted to subfolders under the current working directory.

Optional bucket override:

generate-item-metadata EO.XXX.YYY.ZZZ --bucket-name usergenerated-proposal-eo.xxx.yyy.zzz

Compatibility command (run from the project root):

python generate_item_metadata.py EO.XXX.YYY.ZZZ

Compatibility command with collection subfolder:

python generate_item_metadata.py EO.XXX.YYY.ZZZ --collection-path toto

Programmatic Usage

You can also use ItemGenerator directly in your Python scripts:

from usergenerated import ItemGenerator

# Create a generator for your collection
generator = ItemGenerator("EO.XXX.YYY.ZZZ")

# Generate Item metadata
generator.run()

# Resolve collection from ./toto/EO.XXX.YYY.ZZZ
generator_with_path = ItemGenerator("EO.XXX.YYY.ZZZ", collection_path="toto")
generator_with_path.run()

# With optional bucket name override
generator_with_bucket = ItemGenerator("EO.XXX.YYY.ZZZ", override_bucket_name="custom-bucket")
generator_with_bucket.run()

# Also write a reordered/normalised copy to output/collection_reordered.json (off by default)
generator_reordered = ItemGenerator("EO.XXX.YYY.ZZZ", save_reordered_collection=True)
generator_reordered.run()

Generated output is written to:

<COLLECTION_ID>/metadata/items/*.json

The ItemGenerator class is the primary API for programmatic use. The CLI command is a thin wrapper around this class.

Regenerate Item Metadata in S3

Use ItemGeneratorS3 to regenerate STAC Item metadata from bucket data while restricting updates to Item metadata only. This is useful when you need to update item generation logic or fix generated metadata without modifying data files in S3.

CLI Usage

Primary command (regenerate items in S3 for your collection):

regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ

This command:

1. Mirrors collection data from S3 to a local folder.
2. Regenerates STAC Item metadata locally with your configured settings.
3. Backs up existing S3 items by default.
4. Replaces only the metadata/items prefix in S3 (never modifies data files).

Optional dry-run (validate without modifying S3):

regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --dry-run

Optional bucket name override:

regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --bucket-name custom-bucket-name

Optional local mirror folder (default: tmp_structure_mirror):

regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --local-root "./my_mirror"

Disable backup:

regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --no-backup-items

Skip the ready_for_transfer safety check (see Safety Behavior; enabled by default):

regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --no-use-ready-for-transfer-flag

Demonstration script usage (env-first configuration via .env/environment variables, see script_s3_regenerate_item_metadata.py for the full list of S3_* variables):

python script_s3_regenerate_item_metadata.py

This script is intentionally a thin example of programmatic ItemGeneratorS3 usage. For routine operations, prefer the regenerate-item-metadata-s3 console command.

The demo script additionally supports USE_READY_FOR_TRANSFER_FLAG (or the use_ready_for_transfer_flag keyword argument to run_regenerate_item_metadata) to toggle the ready_for_transfer safety check described below — see Safety Behavior. The primary console command exposes the same toggle via --use-ready-for-transfer-flag/--no-use-ready-for-transfer-flag (shown above).

Programmatic Usage

You can also use ItemGeneratorS3 directly in your Python scripts:

from usergenerated import ItemGeneratorS3

# Create a generator for your collection
generator = ItemGeneratorS3(
    "EO.XXX.YYY.ZZZ",
    dry_run=True,  # Validate without modifying S3
)

# Regenerate Item metadata
generator.run()

# With explicit bucket name
generator_with_bucket = ItemGeneratorS3(
    "EO.XXX.YYY.ZZZ",
    bucket_name="custom-bucket",
    backup_items=True,
)
generator_with_bucket.run()

# Skip the ready_for_transfer safety check (see Safety Behavior below); enabled by default
generator_without_gate = ItemGeneratorS3(
    "EO.XXX.YYY.ZZZ",
    use_ready_for_transfer_flag=False,
)
generator_without_gate.run()

Safety Behavior

The regeneration workflow includes several safety measures:

1. Validates credentials and bucket structure before processing.
2. Mirrors S3 content locally and ensures local processing under <collection_id>/.
3. By default (use_ready_for_transfer_flag=True), requires "ready_for_transfer": true in the mirrored collection's metadata/collection_config.json; aborts with an error if the field is missing or false (this check applies even with --dry-run). Skip this check via --no-use-ready-for-transfer-flag on the CLI, the use_ready_for_transfer_flag=False constructor argument, or USE_READY_FOR_TRANSFER_FLAG=false for the demo script.
4. Regenerates Item metadata locally with existing generation logic.
5. Backs up existing <collection_id>/metadata/items by default (can be disabled).
6. Replaces only <collection_id>/metadata/items in S3 (never modifies data/).
7. With --dry-run, performs local validation and regeneration, then prints planned S3 actions without modifying the bucket.

Mirroring Behavior

The mirroring process downloads real files selectively to improve performance:

1. Downloads real files only for metadata/* and item_config.json keys.
2. Writes per-folder manifest files (usergenerated_s3_folder_manifest.jsonl) for discovered objects.
3. Uses manifest entries during item generation so full non-metadata assets are not required locally.
4. Supports optional placeholder file creation for non-fetched objects (disabled in the S3 regeneration workflow).
5. Logs a summary with discovered, downloaded, and failed object counts, plus periodic progress logs during large scans.

Maintainer Notes

Recent internal refactoring improved testability and packaging workflows.

1. The primary command is now generate-item-metadata <COLLECTION_ID>.
2. generate_item_metadata.py is retained as a compatibility wrapper.
3. Internal logic was modularized into reusable helpers under usergenerated/ for:

• collection path resolution
• item folder discovery by configured granularity
• item config precedence/loading
• simplified and hierarchy-based datetime resolution

4. This split enables unit-heavy testing of pure logic while keeping entry-point orchestration thin.

Optional S3 Upload

Automatic upload is configured at runtime via ItemGenerator constructor parameters or CLI flags (not via module constants).

Programmatic example:

from usergenerated import ItemGenerator

generator = ItemGenerator(
    "EO.XXX.YYY.ZZZ",
    is_upload_s3=True,
    s3_endpoint_url="https://s3.central.data.destination-earth.eu",
    s3_user_generated_bucket_prefix="usergenerated-proposal",
)
generator.run()

CLI example:

generate-item-metadata EO.XXX.YYY.ZZZ \
  --upload-s3 \
  --s3-endpoint-url "https://s3.central.data.destination-earth.eu" \
  --s3-bucket-prefix "usergenerated-proposal"

If upload is enabled, create a .env file at the project root with credentials:

AWS_ACCESS_KEY_ID="<your_access_key_id>"
AWS_SECRET_ACCESS_KEY="<your_secret_access_key>"

For manual upload workflows, see README-s3cmd.md.

S3 utility wrappers are available for programmatic execution around S3Tools:

from script_s3_create_bucket import run_create_bucket
from script_s3_move_bucket_contents import run_move_bucket_contents
from script_s3_upload_file_to_bucket import run_upload_file
from script_s3_upload_folder_to_bucket import run_upload_folder

run_create_bucket(
  aws_access_key_id="<key>",
  aws_secret_access_key="<secret>",
  s3_bucket_name="my-bucket",
)

run_upload_file(
  file_name="EO.XXX.YYY.ZZZ/metadata/collection.json",
  object_name="EO.XXX.YYY.ZZZ/metadata/collection.json",
  aws_access_key_id="<key>",
  aws_secret_access_key="<secret>",
  s3_bucket_name="my-bucket",
)

run_upload_folder(
  folder_path="EO.XXX.YYY.ZZZ/metadata/items",
  target_path="EO.XXX.YYY.ZZZ/metadata/items",
  aws_access_key_id="<key>",
  aws_secret_access_key="<secret>",
  s3_bucket_name="my-bucket",
)

run_move_bucket_contents(
  aws_access_key_id="<key>",
  aws_secret_access_key="<secret>",
  s3_bucket_name="my-bucket",
  s3_target_prefix="archive",
)

S3 Folder Upload Progress

When using run_upload_folder(...) or S3Tools.upload_folder_to_s3(...), uploads now display progress bars:

1. A global Total files bar showing overall folder upload progress.
2. Per-file bars (bytes transferred) for active uploads.

Current behavior:

1. Uploads run concurrently with a bounded worker pool.
2. Per-file progress bars are capped to a visible range and positions are reused for large uploads.
3. usergenerated_s3_folder_manifest.jsonl files and symlinks are skipped from upload discovery.
4. A final summary logs discovered, uploaded, and failed file counts.

These wrappers accept explicit parameters first and only fall back to environment variables when a parameter is omitted.

Regenerate Item Metadata: Use the CLI command for routine operations (see Regenerate Item Metadata in S3):

regenerate-item-metadata-s3 EO.XXX.YYY.ZZZ --dry-run

The script_s3_regenerate_item_metadata.py file is kept as a small demonstrator of ItemGeneratorS3 programmatic usage. Use the CLI command or the ItemGeneratorS3 class directly for real workflows.

Copy a Collection Between Buckets

script_s3_copy_source_bucket_collection_to_target_bucket_collection.py copies a collection folder from a source S3 bucket/account to a target S3 bucket/account, using usergenerated.s3.s3_collection_copier.CollectionCopier. It:

• Detects the source bucket's BucketLayout (COLLECTION_ROOT or FLAT_ROOT) and normalizes the target to always be <collection_id>/... (COLLECTION_ROOT), fixing up FLAT_ROOT sources in the process.
• Never modifies the source bucket.
• Is safely re-runnable: objects already present in the target (by key) are skipped, so an interrupted transfer can just be re-run to continue where it left off.
• Streams each object directly from source to target (no local disk buffering), since source and target buckets are assumed to use different credentials/accounts.

Configure both buckets in .env (the source variables are the same ones used elsewhere in this project; target variables are TARGET_-prefixed):

# Source bucket (existing variables, unchanged)
S3_ENDPOINT_URL="https://s3.central.data.destination-earth.eu"
S3_BUCKET_NAME="my-source-bucket"
AWS_ACCESS_KEY_ID="<source_access_key_id>"
AWS_SECRET_ACCESS_KEY="<source_secret_access_key>"

# Target bucket
TARGET_S3_ENDPOINT_URL="https://s3.central.data.destination-earth.eu"
TARGET_S3_BUCKET_NAME="my-target-bucket"
TARGET_AWS_ACCESS_KEY_ID="<target_access_key_id>"
TARGET_AWS_SECRET_ACCESS_KEY="<target_secret_access_key>"

S3_COLLECTION_ID="EO.XXX.YYY.ZZZ"
S3_FRESH_START="false"  # set true to delete the target's existing '<collection_id>/' prefix first
S3_DRY_RUN="false"      # set true to preview the copy plan without modifying the target

Then run:

python script_s3_copy_source_bucket_collection_to_target_bucket_collection.py

S3_FRESH_START is the only way to trigger the destructive "clean the target path first" step (e.g. for a first run against a target that already has stray/partial contents); it is never triggered automatically. Omit it (or leave it false) for the default incremental, resume-friendly sync.

Folder uploads use object keys relative to the selected local folder. For example, uploading ./EO.XXX.YYY.ZZZ/metadata/items with no target_path uploads files like a.json and nested/b.json at the bucket root. To place files under a collection or metadata prefix in the bucket, pass that prefix explicitly as target_path.

Verify a Collection Copy

script_s3_verify_collection_copy.py is a read-only smoke test to run right after the copy script above completes: it uses usergenerated.s3.s3_collection_copy_verifier.CollectionCopyVerifier to list the collection's objects in both buckets and confirms the target has exactly the same relative keys as the source, with matching sizes for every key. Neither bucket is modified.

It reuses the same .env variables as the copy script above (no new variables needed):

python script_s3_verify_collection_copy.py

The script exits 0 and prints Collection copy verification passed: True when the target matches the source. On a mismatch CollectionCopyVerifier.run() raises S3OperationError whose message summarises the missing, unexpected, and/or size-mismatched keys (sampled, up to 20 per category); the script reports that one line and exits 1.

Similarly, CollectionCopier.run() (the copy script above) raises S3OperationError — naming the failed-object count and a sample of failed keys — if any object fails to copy. Objects that did copy are left in place, so re-running resumes from where it stopped.

S3 helper code lives under usergenerated/s3/.

Promote a Collection (regenerate → copy → verify → read, in one call)

usergenerated.s3.collection_promoter.promote_collection (also from usergenerated import promote_collection) runs the whole "a collection has been staged in a source bucket; validate it, copy it to a target bucket, and hand me its items" workflow as a single call, in strict order:

1. ItemGeneratorS3(...).run() — regenerate STAC item metadata in the source bucket (source credentials; the ready_for_transfer safety gate stays on; data/ is untouched).
2. CollectionCopier(...).run() — copy the collection source bucket → target bucket.
3. CollectionCopyVerifier(...).run() — confirm the target matches the source key-for-key.
4. Build a PromotedCollection read accessor over the target copy and call on_ready(handle) exactly once.

on_ready is a callback you provide. Its argument exposes just two methods:

• handle.read_collection() -> dict — the target copy's collection.json (wraps get_collection_from_s3).
• handle.iter_items() -> Iterator[dict] — the target copy's STAC Items, streamed one at a time (wraps iter_collection_items_from_s3; lazy — consume it inside the callback).

Any domain-specific downstream work (database load, re-publish, indexing) stays entirely in your on_ready; promote_collection owns no datastore and never returns a status — a normal return means every stage passed and on_ready ran.

• on_ready runs only after copy and verify have both succeeded. Any earlier failure raises (S3OperationError / ConfigError / CredentialsError / FileNotFoundError, and FileExistsError from regeneration) and on_ready is never called.
• dry_run=True runs regeneration/validation only (via ItemGeneratorS3(dry_run=True)); the copier and verifier are never constructed and on_ready is never called.
• fresh_start=True passes through to CollectionCopier, deleting the target <collection_id>/ prefix before copying. The source bucket is never modified.
• Idempotent / resumable: re-running after a failure repeats the skip-aware copy, re-verifies, and re-invokes on_ready. Side effects are only on the source (item regen, plus a timestamped items backup when backup_items=True) and the target (the copy).

from usergenerated import PromotedCollection, promote_collection


def on_ready(collection: PromotedCollection) -> None:
    stac = collection.read_collection()
    item_count = sum(1 for _ in collection.iter_items())
    print(f"{stac['id']}: {item_count} item(s) ready in the target bucket")


promote_collection(
    collection_id="EO.XXX.YYY.ZZZ",
    source_endpoint_url="https://s3.central.data.destination-earth.eu",
    source_bucket_name="my-source-bucket",
    source_aws_access_key_id="<source_access_key_id>",
    source_aws_secret_access_key="<source_secret_access_key>",
    target_endpoint_url="https://s3.central.data.destination-earth.eu",
    target_bucket_name="my-target-bucket",
    target_aws_access_key_id="<target_access_key_id>",
    target_aws_secret_access_key="<target_secret_access_key>",
    on_ready=on_ready,
)

script_s3_promote_collection.py is a runnable .env-driven demo whose on_ready logs the promoted collection's id and item count. It reuses the same variables as the copy and regenerate scripts (S3_COLLECTION_ID, S3_ENDPOINT_URL, S3_BUCKET_NAME, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, TARGET_S3_ENDPOINT_URL, TARGET_S3_BUCKET_NAME, TARGET_AWS_ACCESS_KEY_ID, TARGET_AWS_SECRET_ACCESS_KEY, S3_DRY_RUN, S3_FRESH_START), plus two optional keys read only by this script: S3_LOCAL_ROOT (default tmp_structure_mirror) and S3_BACKUP_ITEMS (default true).

python script_s3_promote_collection.py

Manipulate Collection Config in S3

script_s3_collection_config_manipulation.py demonstrates the helpers in usergenerated.s3.s3_collection_config_tools for reading and updating a collection's collection_config.json directly in S3, without a local mirror. Unlike the verify script above, it mutates the live bucket. For a given collection it:

1. Resolves the config key with resolve_collection_config_key and prints the full collection_config.json.
2. Reads the ready_for_transfer flag via get_ready_for_transfer.
3. Sets ready_for_transfer to false via set_ready_for_transfer.
4. Adds an ad-hoc last_transferred field (current UTC time, ISO 8601) via update_collection_config.

Setting ready_for_transfer to false engages the regenerate-item-metadata-s3 safety gate described in Safety Behavior — a subsequent regeneration run will refuse to proceed until the flag is set back to true (or the check is explicitly skipped).

For programmatic use the same module exposes two reusable building blocks so a caller does not have to chain layout detection and a hand-written read-modify-write:

• resolve_collection_config_key(s3_client, bucket_name, collection_id) — locate collection_config.json in the bucket without handling BucketLayout yourself.
• update_collection_config(s3_client, bucket_name, key, changes) — one read-modify-write for an arbitrary set of fields. changes is either a mapping merged into the config (use the exported REMOVE sentinel as a value to delete a key) or a callback that mutates the dict in place. Keys you do not touch keep their value, order, and the file's indent=4 formatting; it raises ConfigError if the object is missing or not a JSON object and S3OperationError on an S3 read/write failure. set_ready_for_transfer is a one-line wrapper over it.

Configure the target bucket in .env (the same single-bucket variables used by the regeneration demo script):

S3_ENDPOINT_URL="https://s3.central.data.destination-earth.eu"
S3_BUCKET_NAME="my-bucket"
AWS_ACCESS_KEY_ID="<access_key_id>"
AWS_SECRET_ACCESS_KEY="<secret_access_key>"
S3_COLLECTION_ID="EO.XXX.YYY.ZZZ"

Then run:

python script_s3_collection_config_manipulation.py

Report Bucket Information

script_s3_bucket_information.py demonstrates usergenerated.s3.s3_bucket_info.get_bucket_information, which makes a single paginated sweep of a bucket (or a prefix within it) and prints a JSON report: file count, total and average size, the 5 largest files, oldest/most-recent modification times, a count+size breakdown by top-level folder, a generated_at timestamp, and a sample file listing capped at 50 entries (with a sample_truncated flag) so you never get thousands of rows back. Every size in the report (total_size_bytes, average_file_size_bytes, and each size_bytes in largest_files, top_level_prefixes, and sample_files) has a ..._human/size_human counterpart, e.g. "total_size_human": "94.2 MB". It is read-only — the bucket is never modified.

Writing the report to output/<YYYYMMDDTHHMMSS>_Bucket_Information.json (the timestamp matches generated_at in the JSON) is off by default — the report is only printed to stdout. Turn it on with either S3_WRITE_OUTPUT_FILE="true" in .env, or write_output_file=True when calling run_get_bucket_information programmatically; the output/ directory is created if needed.

Configure the bucket in .env (the same single-bucket variables used by the config manipulation script above); S3_PREFIX, S3_BUCKET_INFO_SAMPLE_LIMIT, S3_WRITE_OUTPUT_FILE, and S3_BUCKET_INFO_OUTPUT_DIR are optional:

S3_ENDPOINT_URL="https://s3.central.data.destination-earth.eu"
S3_BUCKET_NAME="my-bucket"
AWS_ACCESS_KEY_ID="<access_key_id>"
AWS_SECRET_ACCESS_KEY="<secret_access_key>"
S3_PREFIX=""                        # optional: scope the report to one folder/collection
S3_BUCKET_INFO_SAMPLE_LIMIT="50"    # optional: cap on the sample file listing
S3_WRITE_OUTPUT_FILE="false"        # optional: set true/1/yes/on to also write the JSON report to disk
S3_BUCKET_INFO_OUTPUT_DIR="output"  # optional: where the timestamped JSON report is written

Then run:

python script_s3_bucket_information.py

Copy Collection Metadata Locally

script_s3_get_metadata.py demonstrates usergenerated.s3.s3_collection_metadata_downloader.download_collection_metadata, which copies a collection's metadata/ folder from S3 to a local destination. The source bucket's layout (whether it holds <collection_id>/metadata/ or a top-level metadata/) is detected automatically, so the caller never needs to know or check it beforehand — the result always lands under a folder named for the collection ID, e.g. <destination_root>/EO.XXX.YYY.ZZZ/metadata. It is read-only with respect to S3 — nothing in the source bucket is modified.

The destination root is configurable via S3_METADATA_DESTINATION_ROOT (or the destination_root argument when calling run_get_metadata programmatically); when unset, it defaults to a fresh timestamped folder under the OS temp directory (/tmp on Linux, %TEMP% on Windows), e.g. <os-temp-dir>/usergenerated_tmp/20260824T101500/EO.XXX.YYY.ZZZ/metadata.

Objects are downloaded concurrently — one worker thread per object, bounded by the max_workers argument (default MAX_METADATA_DOWNLOAD_WORKERS = 16, capped at 32). The single s3_client is shared across those threads, so build it with a matching connection pool: create_s3_client(..., max_pool_connections=MAX_METADATA_DOWNLOAD_WORKERS) (botocore defaults to 10). If any object fails to download the whole call raises S3OperationError (a RuntimeError subclass) rather than returning a partial folder; once a systemic problem has failed ~20 objects it stops submitting new downloads instead of grinding through every remaining key.

For callers that only need the files for the duration of a block — parse collection.json, stream items/ somewhere, done — use the downloaded_collection_metadata context manager instead of calling download_collection_metadata directly; on exit it removes the <collection_id>/ subtree it downloaded, and if it also generated the timestamped destination root (no explicit destination_root given) it removes that too, so nothing is left under /tmp. An explicit destination_root you pass in is otherwise left in place:

with downloaded_collection_metadata(s3_client, bucket, collection_id) as metadata_folder:
    collection = get_stac_collection(metadata_folder)
    for item in iter_items(metadata_folder):
        ...
# the timestamped <os-temp-dir>/usergenerated_tmp/<...> directory is gone here

The same module also provides small supporting functions for reading the downloaded metadata: get_stac_collection/get_collection_config load collection.json/collection_config.json as dicts, and iter_items/iter_item_batches stream STAC Items out of the downloaded items/ folder — one item at a time, or in fixed-size batches — rather than loading every item into memory at once. This matters for collections with very large item counts (hundreds of thousands), where materializing every item as a single list would be memory-intensive. The batch size defaults to METADATA_DOWNLOADER_DEFAULT_ITEM_BATCH_SIZE (500) and is configurable via S3_METADATA_ITEM_BATCH_SIZE (or the item_batch_size argument to run_get_metadata).

If you only need to walk the items once and don't need them to persist, iter_collection_items_from_s3 / iter_collection_item_batches_from_s3 (and get_collection_from_s3 for collection.json) stream STAC Items straight from the bucket with no local copy written and no cleanup to do — a drop-in for iter_items / iter_item_batches / get_stac_collection. The bucket layout is detected automatically, item bodies are fetched with a bounded concurrent prefetch (max_workers defaults to MAX_METADATA_DOWNLOAD_WORKERS, capped at 32), and any fetch or parse failure raises S3OperationError rather than yielding a partial result. Keep download_collection_metadata for when the files must stay on disk or be read more than once. Setting S3_METADATA_STREAM_ITEMS to a truthy value (or passing stream_items=True to run_get_metadata) runs script_s3_get_metadata.py on this no-download path.

Configure the collection and bucket in .env (the same single-bucket variables used by the config manipulation script above); S3_METADATA_DESTINATION_ROOT, S3_METADATA_ITEM_BATCH_SIZE, and S3_METADATA_STREAM_ITEMS are optional:

S3_COLLECTION_ID="EO.XXX.YYY.ZZZ"
S3_ENDPOINT_URL="https://s3.central.data.destination-earth.eu"
S3_BUCKET_NAME="my-bucket"
AWS_ACCESS_KEY_ID="<access_key_id>"
AWS_SECRET_ACCESS_KEY="<secret_access_key>"
S3_METADATA_DESTINATION_ROOT=""  # optional: local folder to copy metadata/ into (default: <os-temp-dir>/usergenerated_tmp/<timestamp>)
S3_METADATA_ITEM_BATCH_SIZE=""  # optional: items per batch when streaming items/ (default: 500)
S3_METADATA_STREAM_ITEMS=""  # optional: truthy = skip the local download, stream items straight from S3

Then run:

python script_s3_get_metadata.py

Inspect a Deployed Collection

A collection is deployed once it is served by the DestinE Data Lake HDA STAC API at https://hda.data.destination-earth.eu/stac/v2/collections. This is the check that comes after a collection has been promoted to the target bucket and picked up by DEDL.

Some collections are protected by a role and are simply absent for anonymous callers, so script_deployed_information.py always makes both attempts and reports them side by side:

• anonymous — a plain request, no credentials.
• authenticated — a request carrying a DEDL access token as an Authorization: Bearer header. The token comes from a DESP username/password via the destinelab token exchange, and makes the collections your account is entitled to see visible.

From the two outcomes it derives:

• deployed — either attempt found the collection.
• requires_authentication — only the authenticated attempt found it, i.e. the collection is deployed but protected.

The script is read-only: it issues HTTP GETs only, and writes nothing to the STAC API or to S3. The DESP password and the DEDL token never appear in the report or in the logs.

Install the optional dependency

The authenticated attempt needs destinelab, which ships as an optional extra so anonymous lookups work on a plain install:

uv sync --extra deployed

Without it (or without DESP credentials), the authenticated attempt is reported as skipped and the run still succeeds with the anonymous result.

Configuration

Add the keys you need to .env:

DEPLOYED_COLLECTION_ID="EO.XXX.YYY.ZZZ"
DESP_USERNAME="<desp_username>"  # optional: omit to run the anonymous lookup only
DESP_PASSWORD="<desp_password>"  # optional: omit to run the anonymous lookup only
DEPLOYED_STAC_BASE_URL=""  # optional: STAC API base URL (default: https://hda.data.destination-earth.eu/stac/v2)
DEPLOYED_AUTHENTICATE=""  # optional: falsy = skip the authenticated attempt entirely (default: true)
DEPLOYED_OUTPUT_DIR=""  # optional: folder for the output file (default: output)
DEPLOYED_WRITE_OUTPUT_FILE=""  # optional: falsy = skip writing the collection file (default: written)

The output file contains only the STAC Collection — not the surrounding report — so it can be diffed straight against a local metadata/collection.json. Before writing, the document is loaded into PySTAC and validated:

• Normalized — the file is PySTAC's own serialization, so its field ordering is standardized rather than whatever the server happened to return, and successive runs produce diffable files. Extension and vendor fields (dedl:*, cube:*, sci:*, …) are preserved.
• Validated — the result is reported in the console report as stac_valid and stac_validation_error, and the report also gains the output_file path. Validation is reported, not enforced: a deployed collection that fails validation is still written, since it is not yours to fix and is exactly the file worth inspecting. A document PySTAC cannot read as a Collection at all raises CollectionValidationError.

The filename carries a filename-safe form of the collection id (every run of non-alphanumeric characters collapses to a single underscore, so EO.XXX.YYY.ZZZ becomes EO_XXX_YYY_ZZZ), so several collections can share one output folder:

output/20260904T164512_EO_XXX_YYY_ZZZ_Deployed_Collection.json

When both attempts found the collection the authenticated body is written (it is the only one that can see a protected collection; for a public one the two are identical). Nothing is written when neither attempt found it.

Then run:

python script_deployed_information.py

Example output for a deployed but role-protected collection (the full STAC Collection body is included under each successful attempt, truncated here):

{
    "collection_id": "EO.XXX.YYY.ZZZ",
    "base_url": "https://hda.data.destination-earth.eu/stac/v2",
    "anonymous": {
        "attempted": true,
        "found": false,
        "status_code": 404,
        "error": "Collection 'EO.XXX.YYY.ZZZ' is not visible to this caller (HTTP 404).",
        "collection": null
    },
    "authenticated": {
        "attempted": true,
        "desp_username": "user@example.org",
        "roles": ["..."],
        "found": true,
        "status_code": 200,
        "error": null,
        "collection": { "type": "Collection", "id": "EO.XXX.YYY.ZZZ", "...": "..." }
    },
    "deployed": true,
    "requires_authentication": true,
    "generated_at": "2026-09-04T16:23:14.859202+00:00"
}

Programmatic Usage

The helpers behind the script live in usergenerated.deployed and can be used directly:

from usergenerated.deployed.auth import authenticate_dedl
from usergenerated.deployed.stac_api import get_collection, list_collection_ids

# Anonymous - only public collections are visible.
lookup = get_collection("EO.ESA.DAT.SENTINEL-2.MSI.L1C")
print(lookup.found, lookup.status_code)

# Authenticated - protected collections the account is entitled to see become visible.
authentication = authenticate_dedl("<desp_username>", "<desp_password>")
protected = get_collection("EO.XXX.YYY.ZZZ", token=authentication.token)
print(authentication.roles, protected.collection)

# Every collection id visible to this caller.
print(len(list_collection_ids(token=authentication.token)))

get_collection returns a CollectionLookup (found, status_code, collection, error). A collection that is not visible is reported as found=False with the HTTP status - not an exception. Transport failures, unexpected statuses, and unparseable bodies raise DeployedApiError; missing DESP credentials or an unusable token raise CredentialsError; a missing destinelab install raises ConfigError telling you to run uv sync --extra deployed.

Runtime Initialization

Logging initialization is now explicit and idempotent.

1. Entry-point scripts call setup_logging() from usergenerated.logging_config at startup.
2. Environment loading (load_dotenv) is also called explicitly at startup.
3. Importing modules should not configure logging implicitly.

This keeps behavior stable regardless of import order and avoids duplicate log handlers.

Windows

The package is tested on Linux and Windows (CI runs the suite on both). A few platform notes:

• Temp directory. Features that mirror or download from S3 (regenerate-item-metadata-s3, script_s3_get_metadata.py) use the OS temp directory (%TEMP% on Windows) for their working copy, not a hardcoded /tmp. Override it with S3_METADATA_DESTINATION_ROOT where supported.

• Content-based MIME sniffing. itemhelper.guess_mime_type_advanced() needs the optional magic extra (uv sync --extra magic / pip install "usergenerated[magic]") and a native libmagic, which is not present on a default Windows install. Nothing in the standard workflow calls it — media types are inferred from the file extension.

• Symlinks in data/. Creating symlinks on Windows needs SeCreateSymbolicLinkPrivilege (Developer Mode or an elevated shell). The tools skip symlinks on upload regardless.

• S3 keys with unusual names. Keys containing characters that are illegal in Windows filenames (: * ? " < > |), or that differ only in case, are sanitised to a safe local name when mirrored; the original key is preserved in the per-folder manifest.

• Long paths (the 260-character limit). Windows refuses to create a path longer than 260 characters unless long-path support is switched on. This is reachable in practice: an item folder name embeds the whole collection id (<collection_id>_<start>_<end>), so a deep working directory plus a long collection id can cross the limit while mirroring. The tools detect this before creating anything and fail with a message naming the path and its length, rather than the misleading WinError 3: The system cannot find the path specified. Fix it by running from a shallower directory, passing a shorter --local-root / S3_METADATA_DESTINATION_ROOT, or enabling long paths system-wide:

  # Run as Administrator, then reboot.
  Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem' `
      -Name LongPathsEnabled -Value 1
  

• Collection folder name case. The Windows filesystem is case-insensitive, so a folder named eo.xxx.yyy.zzz would otherwise satisfy a request for EO.XXX.YYY.ZZZ and then fail for a colleague on Linux. Collection ids are case-sensitive, so this is reported as an error on every OS — rename the folder to match the id exactly.

• Console output. Log records and progress bars are written as UTF-8 with a safe fallback, so a non-ASCII filename or collection description does not break a legacy cmd.exe console.

Exit codes and failure output

Every console command (generate-item-metadata, regenerate-item-metadata-s3) and root script_s3_*.py shares one convention:

Exit code	Meaning
0	success
1	an expected failure (bad config, missing credentials, a missing file or folder you named, invalid collection.json, a failed S3 operation) — reported as a single ERROR log line, no traceback; or an unexpected internal error, which keeps its traceback
2	command-line usage error (bad or missing arguments)

Troubleshooting

1. Generation fails with folder-level errors: Set item_folder_level to match your actual data/ layout.

2. Missing item_config.json errors: Set item_config_optional to true if item config files are intentionally omitted.

3. Collection id validation errors: Ensure root folder name and collection_config.json > id represent the same collection id.

4. Upload credential errors: Ensure .env contains valid AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY when upload is enabled.

External References

1. DestinE Data Lake - Promote User Data to become DestinE data
2. STAC Collection specification
3. STAC Item specification