🪐 NASA PDS Registry repository

This repository is an umbrella for the PDS registry application. The application is composed of multiple sub-components, each of them managed in their own repository and integrated here:

• registry-loader: tools to load PDD4 producs to the registry
• opensearch: the famoua data store and search engine which performaces we praise.s
• registry-sweepers: suite of script consolidating the PDS4 product descriptions in OpenSearch.
• registry-api: the PDS Search API service.
• registry-ref-data: some reference datsets that we use for our application integration tests.

Users and administrator should use the documentation published on http://nasa-pds.github.io/registry

This repository specifically contains these PDS registry application resources:

• the source for the user/administrator documentation, see docs folder
• docker compose script to start/test the full application with its required components, see https://github.com/NASA-PDS/registry/blob/main/docker/README.md for details. This also includes integration tests (in docker/postman). The integration test datasets are maintained in NASA-PDS/registry-ref-data repository.
• For developers: See Integration Testing Guide for instructions on adding and running integration tests.
• utility scripts for registry management and reporting, see scripts folder

Code of Conduct

All users and developers of the NASA-PDS software are expected to abide by our Code of Conduct. Please read this to ensure you understand the expectations of our community.

Scripts

Registry Status Report Generator

Generate CSV reports for missing and staged products in the PDS Registry:

# Generate reports and commit/push to GitHub (default)
./scripts/generate_registry_status_reports.py

# Generate reports only, without committing
./scripts/generate_registry_status_reports.py --no-commit

Purpose: Generates reports for missing and staged products in the registry.

Output: Creates four CSV files in docs/status/:

• missing_bundles_in_registry.csv - Missing Product_Bundle records
• missing_collections_in_registry.csv - Missing Product_Collection records
• staged_bundles_in_registry.csv - Staged Product_Bundle records
• staged_collections_in_registry.csv - Staged Product_Collection records

By default, the script commits and pushes these files to GitHub. Use --no-commit to disable this.

Requirements:

• Python 3.12 or higher
• pds-registry-client and PDS Registry credentials
• See scripts/README.md for full configuration details

Utilities

• Treks

  • To deploy the package run one of these commands from the root directory: pip install . for users pip install -e '.[dev]' for developers
    • This package is also hosted on the "cheeseshop" and can be installed with pip install pds.registry
  • The Treks utilities can be used to create pds4 labels for the layers given in the Treks API
  • Run the command: create-treks-pds4

• GeoSTAC - Lola

  • To deploy the package run one of these commands from the root directory: pip install . for users pip install -e '.[dev]' for developers
    • This package is also hosted on the "cheeseshop" and can be installed with pip install pds.registry
  • The GeoSTAC utilities can be used to create pds4 labels for the Lola point clouds they host
  • This command needs LOLA GDR data to be loaded in the registry in order to connect the lid references
    • This data can be found here: https://pds-geosciences.wustl.edu/lro/lro-l-lola-3-rdr-v1/lrolol_1xxx/data/lola_gdr/cylindrical/float_img/
  • Run the command: create-lola-pds4

Development

To develop this project, use your favorite text editor, or an integrated development environment with Python support, such as PyCharm.

Contributing

For information on how to contribute to NASA-PDS codebases please take a look at our Contributing guidelines.

Installation

Install in editable mode and with extra developer dependencies into your virtual environment of choice:

pip install --editable '.[dev]'

Make a baseline for any secrets (email addresses, passwords, API keys, etc.) in the repository:

detect-secrets scan . \
    --all-files \
    --disable-plugin AbsolutePathDetectorExperimental \
    --exclude-files '\.secrets..*' \
    --exclude-files '\.git.*' \
    --exclude-files '\.mypy_cache' \
    --exclude-files '\.pytest_cache' \
    --exclude-files '\.tox' \
    --exclude-files '\.venv' \
    --exclude-files 'venv' \
    --exclude-files 'dist' \
    --exclude-files 'build' \
    --exclude-files '.*\.egg-info' > .secrets.baseline

Review the secrets to determine which should be allowed and which are false positives:

detect-secrets audit .secrets.baseline

Please remove any secrets that should not be seen by the public. You can then add the baseline file to the commit:

git add .secrets.baseline

Then, configure the pre-commit hooks:

pre-commit install
pre-commit install -t pre-push
pre-commit install -t prepare-commit-msg
pre-commit install -t commit-msg

These hooks then will check for any future commits that might contain secrets. They also check code formatting, PEP8 compliance, type hints, etc.

👉 Note: A one time setup is required both to support detect-secrets and in your global Git configuration. See the wiki entry on Secrets to learn how.

Documentation

The project uses Sphinx to build its documentation. PDS' documentation template is already configured as part of the default build. You can build your projects docs with:

cd docs
make html

The generated documnentation can be found in directory docs/build/html

CI/CD

The template repository comes with our two "standard" CI/CD workflows, stable-cicd and unstable-cicd. The unstable build runs on any push to main (± ignoring changes to specific files) and the stable build runs on push of a release branch of the form release/<release version>. Both of these make use of our GitHub actions build step, Roundup. The unstable-cicd will generate (and constantly update) a SNAPSHOT release. If you haven't done a formal software release you will end up with a v0.0.0-SNAPSHOT release (see NASA-PDS/roundup-action#56 for specifics).