rda-python-dsarch 3.0.2

RDA Python package for utility program dsarch archive RDA data

dsarch

Python project to archive and maintain datasets for the NSF NCAR Geoscience Data Exchange (GDEX).

The user guide for this utility tool can be viewed at: User guide.

Code overview

The package is built around three Python modules in src/rda_python_dsarch/:

• dsarch.py — Command-line entry point. Defines DsArch, the action class that inherits from PgArch and PgMeta. Its two driver methods read_parameters() and start_actions() parse the CLI and dispatch to the matching handler (archive, get, set, move, delete, restore) for web, saved, help, and Quasar-backup files, plus dataset, group, and version metadata. After the primary action, it also performs cross-cutting follow-up: dataset period updates, file-count resets (-WN/-WM/-RT), final logging, and optional email notification.

• pg_arch.py — Shared state and helpers. Defines the PgArch mixin (extends PgOPT, PgCMD, PgSplit from rda_python_common). Holds the master OPTS table that maps short action codes (e.g. AW, GD, SG, RQ) to their bit flags, names, and write-mode levels; the runtime path cache (RTPATH, webpaths, savedpaths); and group-type/display-order caches. Provides the utility methods used across all actions: SQL condition building, path resolution, file-name validation, and dataset/ group metadata caching.

• pg_meta.py — File-count bookkeeping and metadata-XML queueing. Defines the PgMeta mixin (extends PgCMD, PgSplit). Tracks pending changes to per-dataset/per-group file counts in GCOUNTS (a 13-slot array covering MSS, web-D/N, and saved buckets) and flushes them to RDADB in batch. Maintains META, a queue of metadata operations dispatched to the external tools gatherxml, dcm, rcm, scm, and sml. Also provides switch_logfile() for redirecting per-action log/error output.

The inheritance chain is DsArch -> PgArch -> PgMeta -> PgCMD/PgSplit, so DsArch instances expose every option, path, count, and database helper through a single object.

Environment setup

Create a Python environment first; package installs in the next section run inside whichever environment you activate here.

Option A — Python venv (DECS machines)

python3 -m venv $ENVHOME          # e.g. /glade/u/home/gdexdata/gdexmsenv
source $ENVHOME/bin/activate

Option B — Conda (DAV/Casper)

conda create --prefix $ENVHOME python=3.12   # e.g. /glade/work/gdexdata/conda-envs/pg-gdex
conda activate $ENVHOME

Installing rda-python-dsarch

Pick whichever install mode fits your workflow. All four pull in the transitive dependencies (rda_python_common, rda_python_setuid, rda_python_miscs) automatically.

For local development, clone this repo alongside your project and install it in editable mode so that changes are picked up without re-installing:

git clone https://github.com/NCAR/rda-python-dsarch.git
cd rda-python-dsarch
pip install -e .

To test a specific branch (e.g. an in-progress feature or fix branch), pass -b/--branch to git clone:

git clone -b <branch-name> https://github.com/NCAR/rda-python-dsarch.git
cd rda-python-dsarch
pip install -e .

For a regular (non-editable) install from a checkout:

pip install /path/to/rda-python-dsarch

For a production install on a system that uses the published distribution:

pip install rda_python_dsarch

Setuid Setup

dsarch is executed as the common user PGLOG['COMMONUSER'] (default gdexdata) via the rda_python_setuid setuid mechanism, which is pulled in automatically as a dependency. After pip install above, choose one of the wiring options below.

Note: If rda_python_setuid is already installed and fully set up in your environment, you can skip the compile step (-c/--compile) and the optional pgstart step (-p/--pgstart). The -l/--link step is still required to wire up this package's own setuid program.

Full setuid install (requires sudo access to COMMONUSER)

Run these steps once per environment:

# 1. Compile the pywrapper C binary (once per environment):
pywrapper-install -c|--compile -n|--username gdexdata

# 2. Wire up dsarch as a setuid entry (or use 'all' to link every setuid_* at once):
pywrapper-install -l|--link dsarch
pywrapper-install -l|--link all

# 3. Optionally, install a pgstart_<loginname> binary so <loginname> (any
#    user in the same group as PGLOG['COMMONUSER']) can run commands as
#    themselves.  Run either by PGLOG['ADMINUSER'] (default zji, if it has
#    'sudo -u <loginname>'), or by <loginname> directly:
pywrapper-install -p|--pgstart -n|--username <loginname>

pywrapper-install with no arguments displays the full user guide.

Simple install (no sudo required, runs as current user)

Users who do not need the setuid mechanism can create a direct symlink instead:

pywrapper-install -l|--link dsarch -s|--simple
pywrapper-install -l|--link all -s|--simple   # or link every setuid_* at once

This creates bin/dsarch -> bin/setuid_dsarch and the program runs as the current user with no privilege change.

Update an existing installation (no sudo required)

When the package is upgraded and a new pywrapper.c is bundled, recompile and reinstall all setuid binaries using the existing pgstart_* binaries:

pywrapper-install -u|--update

Setup guide

The shared setuid setup guide is shown automatically if setuid_dsarch is invoked directly before the setuid wrapper has been configured.

Documentation sync

The user guide at gdex-docs-dsarch.readthedocs.io is generated from src/rda_python_dsarch/dsarch.usg. Keep all user-facing content in dsarch.usg — no manual RST editing is required.

When a pull request modifying dsarch.usg is opened, an automated workflow converts it into RST source files and the version number from this repository's pyproject.toml into the gdex-docs-dsarch repository, then opens a pull request from automated-update-branch against its main branch for review.

To publish to Read the Docs:

• Merge that pull request into main to serve the content as the latest version.
• Create a GitHub release in gdex-docs-dsarch to serve the latest release as the stable version.