dsarch
Python project to archive and maintain datasets for the NSF NCAR Geoscience Data Exchange (GDEX).
The user guide for this utility tool can be viewed at: User guide.
Code overview
The package is built around three Python modules in src/rda_python_dsarch/:
-
dsarch.py— Command-line entry point. DefinesDsArch, the action class that inherits fromPgArchandPgMeta. Its two driver methodsread_parameters()andstart_actions()parse the CLI and dispatch to the matching handler (archive, get, set, move, delete, restore) for web, saved, help, and Quasar-backup files, plus dataset, group, and version metadata. After the primary action, it also performs cross-cutting follow-up: dataset period updates, file-count resets (-WN/-WM/-RT), final logging, and optional email notification. -
pg_arch.py— Shared state and helpers. Defines thePgArchmixin (extendsPgOPT,PgCMD,PgSplitfromrda_python_common). Holds the masterOPTStable that maps short action codes (e.g.AW,GD,SG,RQ) to their bit flags, names, and write-mode levels; the runtime path cache (RTPATH,webpaths,savedpaths); and group-type/display-order caches. Provides the utility methods used across all actions: SQL condition building, path resolution, file-name validation, and dataset/ group metadata caching. -
pg_meta.py— File-count bookkeeping and metadata-XML queueing. Defines thePgMetamixin (extendsPgCMD,PgSplit). Tracks pending changes to per-dataset/per-group file counts inGCOUNTS(a 13-slot array covering MSS, web-D/N, and saved buckets) and flushes them to RDADB in batch. MaintainsMETA, a queue of metadata operations dispatched to the external toolsgatherxml,dcm,rcm,scm, andsml. Also providesswitch_logfile()for redirecting per-action log/error output.
The inheritance chain is DsArch -> PgArch -> PgMeta -> PgCMD/PgSplit, so
DsArch instances expose every option, path, count, and database helper
through a single object.
Environment setup
Create a Python environment first; package installs in the next section run inside whichever environment you activate here.
Option A — Python venv (DECS machines)
python3 -m venv $ENVHOME # e.g. /glade/u/home/gdexdata/gdexmsenv
source $ENVHOME/bin/activate
Option B — Conda (DAV/Casper)
conda create --prefix $ENVHOME python=3.12 # e.g. /glade/work/gdexdata/conda-envs/pg-gdex
conda activate $ENVHOME
Installing rda-python-dsarch
Pick whichever install mode fits your workflow. All four pull in the
transitive dependencies (rda_python_common, rda_python_setuid,
rda_python_miscs) automatically.
For local development, clone this repo alongside your project and install it in editable mode so that changes are picked up without re-installing:
git clone https://github.com/NCAR/rda-python-dsarch.git
cd rda-python-dsarch
pip install -e .
To test a specific branch (e.g. an in-progress feature or fix branch), pass
-b/--branch to git clone:
git clone -b <branch-name> https://github.com/NCAR/rda-python-dsarch.git
cd rda-python-dsarch
pip install -e .
For a regular (non-editable) install from a checkout:
pip install /path/to/rda-python-dsarch
For a production install on a system that uses the published distribution:
pip install rda_python_dsarch
Setuid Setup
dsarch is executed as the common user PGLOG['COMMONUSER'] (default
gdexdata) via the rda_python_setuid setuid mechanism, which is pulled
in automatically as a dependency. After pip install above, choose one
of the wiring options below.
Note: If
rda_python_setuidis already installed and fully set up in your environment, you can skip the compile step (-c/--compile) and the optionalpgstartstep (-p/--pgstart). The-l/--linkstep is still required to wire up this package's own setuid program.
Full setuid install (requires sudo access to COMMONUSER)
Run these steps once per environment:
# 1. Compile the pywrapper C binary (once per environment):
pywrapper-install -c|--compile -n|--username gdexdata
# 2. Wire up dsarch as a setuid entry (or use 'all' to link every setuid_* at once):
pywrapper-install -l|--link dsarch
pywrapper-install -l|--link all
# 3. Optionally, install a pgstart_<loginname> binary so <loginname> (any
# user in the same group as PGLOG['COMMONUSER']) can run commands as
# themselves. Run either by PGLOG['ADMINUSER'] (default zji, if it has
# 'sudo -u <loginname>'), or by <loginname> directly:
pywrapper-install -p|--pgstart -n|--username <loginname>
pywrapper-install with no arguments displays the full user guide.
Simple install (no sudo required, runs as current user)
Users who do not need the setuid mechanism can create a direct symlink instead:
pywrapper-install -l|--link dsarch -s|--simple
pywrapper-install -l|--link all -s|--simple # or link every setuid_* at once
This creates bin/dsarch -> bin/setuid_dsarch and the program runs as the
current user with no privilege change.
Update an existing installation (no sudo required)
When the package is upgraded and a new pywrapper.c is bundled, recompile and
reinstall all setuid binaries using the existing pgstart_* binaries:
pywrapper-install -u|--update
Setup guide
The shared setuid setup guide is shown automatically if setuid_dsarch is
invoked directly before the setuid wrapper has been configured.
Documentation sync
The user guide at
gdex-docs-dsarch.readthedocs.io is
generated from src/rda_python_dsarch/dsarch.usg. Keep all user-facing content
in dsarch.usg — no manual RST editing is required.
When a pull request modifying dsarch.usg is opened, an automated workflow
converts it into RST source files and the version number from this repository's
pyproject.toml into the
gdex-docs-dsarch repository, then
opens a pull request from automated-update-branch against its main branch for
review.
To publish to Read the Docs:
- Merge that pull request into
mainto serve the content as thelatestversion. - Create a GitHub release in
gdex-docs-dsarchto serve the latest release as thestableversion.
1 maintainer
Metadata
1 maintainer
Release files for rda-python-dsarch 3.0.14
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| rda_python_dsarch-3.0.14.tar.gz | 152.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| rda_python_dsarch-3.0.14-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 306.6 kB
Release files / rda_python_dsarch-3.0.14.tar.gz
| Download URL | rda_python_dsarch-3.0.14.tar.gz |
|---|---|
| Size | 152.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a499ae16e9aaba1c4bdb0128451fc233204128188c554f7dd55be9e9377a587c
|
|
BLAKE2b-256 checksum How to use checksums |
4a38b1255cfa8d1e96d1829e168700bfe7c5dc96f283045a37901ee47dd2c006
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 14, 2026.
Transparency logRelease files / rda_python_dsarch-3.0.14-py3-none-any.whl
| Download URL | rda_python_dsarch-3.0.14-py3-none-any.whl |
|---|---|
| Size | 154.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f25da53c17afadad4ef77dcfebdccad973218849917132e70d8cb7c353b1d92e
|
|
BLAKE2b-256 checksum How to use checksums |
4f221fb4e301a82f419d93763e761c3b64d278d8dc12659c056de4c886595739
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 14, 2026.
Transparency log
