dsarch

Python project to archive and maintain datasets for the NSF NCAR Geoscience Data Exchange (GDEX).

The user guide for this utility tool can be viewed at: User guide.

Code overview

The package is built around three Python modules in src/rda_python_dsarch/:

  • dsarch.py — Command-line entry point. Defines DsArch, the action class that inherits from PgArch and PgMeta. Its two driver methods read_parameters() and start_actions() parse the CLI and dispatch to the matching handler (archive, get, set, move, delete, restore) for web, saved, help, and Quasar-backup files, plus dataset, group, and version metadata. After the primary action, it also performs cross-cutting follow-up: dataset period updates, file-count resets (-WN/-WM/-RT), final logging, and optional email notification.

  • pg_arch.py — Shared state and helpers. Defines the PgArch mixin (extends PgOPT, PgCMD, PgSplit from rda_python_common). Holds the master OPTS table that maps short action codes (e.g. AW, GD, SG, RQ) to their bit flags, names, and write-mode levels; the runtime path cache (RTPATH, webpaths, savedpaths); and group-type/display-order caches. Provides the utility methods used across all actions: SQL condition building, path resolution, file-name validation, and dataset/ group metadata caching.

  • pg_meta.py — File-count bookkeeping and metadata-XML queueing. Defines the PgMeta mixin (extends PgCMD, PgSplit). Tracks pending changes to per-dataset/per-group file counts in GCOUNTS (a 13-slot array covering MSS, web-D/N, and saved buckets) and flushes them to RDADB in batch. Maintains META, a queue of metadata operations dispatched to the external tools gatherxml, dcm, rcm, scm, and sml. Also provides switch_logfile() for redirecting per-action log/error output.

The inheritance chain is DsArch -> PgArch -> PgMeta -> PgCMD/PgSplit, so DsArch instances expose every option, path, count, and database helper through a single object.

Environment setup

Create a Python environment first; package installs in the next section run inside whichever environment you activate here.

Option A — Python venv (DECS machines)

python3 -m venv $ENVHOME          # e.g. /glade/u/home/gdexdata/gdexmsenv
source $ENVHOME/bin/activate

Option B — Conda (DAV/Casper)

conda create --prefix $ENVHOME python=3.12   # e.g. /glade/work/gdexdata/conda-envs/pg-gdex
conda activate $ENVHOME

Installing rda-python-dsarch

Pick whichever install mode fits your workflow. All four pull in the transitive dependencies (rda_python_common, rda_python_setuid, rda_python_miscs) automatically.

For local development, clone this repo alongside your project and install it in editable mode so that changes are picked up without re-installing:

git clone https://github.com/NCAR/rda-python-dsarch.git
cd rda-python-dsarch
pip install -e .

To test a specific branch (e.g. an in-progress feature or fix branch), pass -b/--branch to git clone:

git clone -b <branch-name> https://github.com/NCAR/rda-python-dsarch.git
cd rda-python-dsarch
pip install -e .

For a regular (non-editable) install from a checkout:

pip install /path/to/rda-python-dsarch

For a production install on a system that uses the published distribution:

pip install rda_python_dsarch

Setuid Setup

dsarch is executed as the common user PGLOG['COMMONUSER'] (default gdexdata) via the rda_python_setuid setuid mechanism, which is pulled in automatically as a dependency. After pip install above, choose one of the wiring options below.

Note: If rda_python_setuid is already installed and fully set up in your environment, you can skip the compile step (-c/--compile) and the optional pgstart step (-p/--pgstart). The -l/--link step is still required to wire up this package's own setuid program.

Full setuid install (requires sudo access to COMMONUSER)

Run these steps once per environment:

# 1. Compile the pywrapper C binary (once per environment):
pywrapper-install -c|--compile -n|--username gdexdata

# 2. Wire up dsarch as a setuid entry (or use 'all' to link every setuid_* at once):
pywrapper-install -l|--link dsarch
pywrapper-install -l|--link all

# 3. Optionally, install a pgstart_<loginname> binary so <loginname> (any
#    user in the same group as PGLOG['COMMONUSER']) can run commands as
#    themselves.  Run either by PGLOG['ADMINUSER'] (default zji, if it has
#    'sudo -u <loginname>'), or by <loginname> directly:
pywrapper-install -p|--pgstart -n|--username <loginname>

pywrapper-install with no arguments displays the full user guide.

Simple install (no sudo required, runs as current user)

Users who do not need the setuid mechanism can create a direct symlink instead:

pywrapper-install -l|--link dsarch -s|--simple
pywrapper-install -l|--link all -s|--simple   # or link every setuid_* at once

This creates bin/dsarch -> bin/setuid_dsarch and the program runs as the current user with no privilege change.

Update an existing installation (no sudo required)

When the package is upgraded and a new pywrapper.c is bundled, recompile and reinstall all setuid binaries using the existing pgstart_* binaries:

pywrapper-install -u|--update

Setup guide

The shared setuid setup guide is shown automatically if setuid_dsarch is invoked directly before the setuid wrapper has been configured.

Documentation sync

The user guide at gdex-docs-dsarch.readthedocs.io is generated from src/rda_python_dsarch/dsarch.usg. Keep all user-facing content in dsarch.usg — no manual RST editing is required.

When a pull request modifying dsarch.usg is opened, an automated workflow converts it into RST source files and the version number from this repository's pyproject.toml into the gdex-docs-dsarch repository, then opens a pull request from automated-update-branch against its main branch for review.

To publish to Read the Docs:

  • Merge that pull request into main to serve the content as the latest version.
  • Create a GitHub release in gdex-docs-dsarch to serve the latest release as the stable version.

Project links

Data verified by PyPI on Aug 24, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (3.0.12)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for zaihuaji from gravatar.com zaihuaji

Credits

PyPI data
Data sourced directly from PyPI's database.

Author: Zaihua Ji

GitHub Statistics

Data verified by PyPI on Aug 24, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

MIT License

Requires

Python >=3.7

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

rda_python_dsarch-3.0.11.tar.gz (147.8 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

rda_python_dsarch-3.0.11-py3-none-any.whl (150.3 kB view details)

Uploaded Python 3

File details

Details for the file rda_python_dsarch-3.0.11.tar.gz.

File metadata

  • Download URL: rda_python_dsarch-3.0.11.tar.gz
  • Upload date:
  • Size: 147.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for rda_python_dsarch-3.0.11.tar.gz
Algorithm Hash digest
SHA256 30833360117ad80553f2a8e3db85552eec5c6d41bcf6190ae5df88d6b9054853
MD5 0904a131b46f6c99e8ae55753b7ada33
BLAKE2b-256 cd6781e33a9180d851c4f3e8024451597f54e7fffb639d9f2fba4d15ba671a77

See more details on using hashes here.

Provenance

The following attestation bundles were made for rda_python_dsarch-3.0.11.tar.gz:

Publisher: publish.yml on NCAR/rda-python-dsarch

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file rda_python_dsarch-3.0.11-py3-none-any.whl.

File metadata

File hashes

Hashes for rda_python_dsarch-3.0.11-py3-none-any.whl
Algorithm Hash digest
SHA256 f8c6f07de5ced2b9363bf34ecee6c8114b22fa310907b666b462cda519f03ac2
MD5 5c2b5a36f58121c5d9a3d4db640ef3c4
BLAKE2b-256 9271a7b5ec60ce609817837a8f4eb0a7aa47b9809c2287c0954f331c9eefe7b1

See more details on using hashes here.

Provenance

The following attestation bundles were made for rda_python_dsarch-3.0.11-py3-none-any.whl:

Publisher: publish.yml on NCAR/rda-python-dsarch

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

3.0.12

2 files

This release

3.0.11 This release

2 files

3.0.10

2 files

3.0.9

2 files

3.0.8

2 files

3.0.7

2 files

3.0.6

2 files

3.0.5

2 files

3.0.3

2 files

3.0.2

2 files

3.0.1

2 files

3.0.0

2 files

2.0.21

2 files

2.0.20

2 files

2.0.19

2 files

2.0.18

2 files

2.0.17

2 files

2.0.16

2 files

2.0.15

2 files

2.0.13

2 files

2.0.12

2 files

2.0.11

2 files

2.0.9

2 files

2.0.5

2 files

2.0.4

2 files

2.0.3

2 files

2.0.2

2 files

2.0.1

2 files

2.0.0

2 files

1.0.32

2 files

1.0.31

2 files

1.0.30

2 files

1.0.29

2 files

1.0.28

2 files

1.0.27

2 files

1.0.25

2 files

1.0.24

2 files

1.0.23

2 files

1.0.22

2 files

1.0.21

2 files

1.0.20

2 files

1.0.19

2 files

1.0.18

2 files

1.0.16

2 files

1.0.15

2 files

1.0.14

2 files

1.0.13

2 files

1.0.12

2 files

1.0.11

2 files

1.0.9

2 files

1.0.7

2 files

1.0.6

2 files

1.0.5

2 files

1.0.4

2 files

1.0.3

2 files

1.0.2

2 files

1.0.1

2 files

1.0.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page