scitex-storage

Research-data storage triage — a read-only, stat-only scan (via fd) that finds the biggest space and inode (file-count) consumers on your disk, safe rotation (referenced-file-aware for build artifacts, compute-node-only tar-in-place for HPC inode hogs), copy-verify-then-remove tiering to NAS, and an opt-in exact-duplicate finder (via fclones).

Full Documentation · pip install scitex-storage

Problem and Solution

Installation

pip install scitex-storage

System dependencies

Three binaries, each a hard runtime dependency of one verb and nothing else. scan and find-duplicates shell out to an established, actively- maintained Rust CLI for their hot path instead of a hand-rolled Python walk/hash — a pure-Python os.walk or hashlib pass is far too slow once you point this at multi-terabyte, multi-million-file storage (this tool is built to scan things like a 4TB NVMe, a multi-TB NAS, or an HDD array). archive/restore need rsync because that is their transport.

# Debian / Ubuntu
sudo apt install fd-find rsync        # fd-find installs the binary as `fdfind`
cargo install fclones                 # no apt package as of this writing

# macOS (Homebrew)
brew install fd fclones               # rsync ships with macOS

# cargo (any platform, if you have a Rust toolchain)
cargo install fd-find fclones

None of these is needed to install scitex-storage — pip has no notion of a system binary. Each is required only when you run its verb, and a missing one raises a MissingSystemDependencyError naming the binary and how to get it, rather than a traceback or a silent slow fallback.

rsync is the easy one to miss, and worth a word on why: nothing in this package spawns it. archive calls scitex-ssh's sync_dir() — an ordinary Python function from an ordinary declared dependency — and the subprocess happens one package away. The PyPI dependency is the adapter; rsync is the thing it adapts, and both need declaring. Your remote host needs rsync too, but a missing one there fails differently (an ssh-side error).

Neither binary is required to install scitex-storage — pip install scitex-storage never needs them, and there is no PyPI package for either (they're not Python libraries). fd IS a hard runtime dependency of scan; fclones IS a hard runtime dependency of find-duplicates. If a binary is missing, the relevant command fails fast with a clear, actionable error (install instructions included) rather than silently falling back to a slow pure-Python walk/hash. scan never needs fclones — it doesn't read file contents at all (see below).

Check what scitex-storage needs system-wide any time with:

python -m scitex_storage._system_deps

GUI plugin (optional)

scitex-storage ships a minimal Django-based GUI plugin (src/scitex_storage/_django/) — a browsable view of scan() results, installable standalone (scitex-storage start-gui) or as a scitex-hub plugin mounted at /storage/, following the same scitex-app/scitex-ui convention as scitex-writer/scitex-scholar/ scitex-todo/figrecipe. It is a first scaffold (proves the mounting contract; not the full disk-treemap UI yet) and is entirely optional:

pip install scitex-storage[gui]
scitex-storage start-gui

Configuration

Copy .env.example to .env (gitignored) at your project root, then edit:

cp .env.example .env
$EDITOR .env

CLI flags always override env vars. The full list of variables (with inline comments) lives in .env.example — both are consumed only by the optional GUI plugin above; the core CLI reads none of them.

Quick Start

# No PATH → inventory ~/.scitex and ~/proj
scitex-storage scan

# A specific tree, ranked by inode count
scitex-storage scan ~/proj --sort files

scitex-storage scan  /home/user/.scitex
=======================================
88.4 GB in 412,003 files across 9 top-level children  (sorted by size)

        SIZE       FILES  CHILD
  ----------  ----------  ------------------------
     71.2 GB     118,204  scholar/
      9.1 GB     251,880  agent-container/
      4.0 GB      12,033  todo/
      ...
  ----------  ----------  ------------------------
     88.4 GB     412,003  TOTAL

Everything is read-only: scan only stats, never reads file contents, never follows symlinked directories, and never moves or deletes anything. It is safe to point at a nearly-full disk or an HPC login node.

Are you about to run out of inodes?

Different question, much cheaper answer. A filesystem can have terabytes free and still fail every write because it is out of inodes — and the jobs that die rarely say so. validate-inodes is one statvfs per path: O(1) rather than O(files), no fd, no login shell, no module load.

# Am I about to hit the wall?
scitex-storage validate-inodes /data/gpfs/projects/punim0264

scitex-storage validate-inodes
===========================
   USED%          USED         TOTAL  MOUNT                 PATH
  ------  ------------  ------------  --------------------  --------------------
    96.2     6,731,073     7,000,000  /data/gpfs            /data/gpfs/projects/punim0264  <-- CRITICAL

That is a real reading (Spartan, 2026-07-17) — and that project was at 70% disk. Space said fine; inodes were four percentage points from every write failing.

On a GPFS independent fileset — how HPC per-project directories are usually carved out — statvfs reports that project's quota, so on those paths this answers the question that actually kills jobs, with no mmlsquota and no login shell.

Verdicts are three-state and never conflated:

Exit codes carry the same distinction, because unattended callers read the exit code and not the table: 0 measured and under --warn-at (default 90%), 1 at/over it, 2 could not look. 2 is deliberately not 0 — a monitor that cannot tell healthy from never read it will report healthy for filesystems it never looked at.

# Cron: alarm on trouble, and alarm just as loudly on blindness.
scitex-storage validate-inodes /data --json || notify "inode check failed ($?)"

# Found something worth checking for exact duplicates? A SEPARATE,
# explicitly opt-in command -- this one DOES read file contents to hash
# them, so use --max-depth on a slow/nearly-full path.
scitex-storage find-duplicates ~/proj/old-scan

3 duplicate groups:

  2 files, 5.0 GB each:
    projects/2022-thesis/dataset.zip
    projects/2022-thesis/dataset (1).zip
  ...

# Move a path ASIDE into a reversible archive instead of deleting it, so a
# rough cleanup call costs a `reclaim-restore`, not lost data. Dry-run by
# default; --yes to act; `reclaim --status` shows the restore rate (the
# accuracy metric). Default archive is an adjacent atomic `.old/<ts>/`
# (tidies, doesn't free inodes); --archive-root on another filesystem frees
# them. Deleting archived data is a separate, later step.
scitex-storage reclaim ./node_modules ./build --yes
scitex-storage reclaim-restore 2026-0717-154500-123456   # undo a run

Architecture

flowchart LR
    A[PATH] -->|os.scandir top level| B[per-child]
    B -->|fd, stat-only, no symlink follow| C[size + inode count]
    C --> D[by_size / by_file_count]
    D --> E[format_report / to_json_dict]
    E --> F[CLI stdout]

    G[DIRECTORY] -->|os.scandir, resolve symlinks| H[referenced set]
    G -->|os.scandir, match --pattern| I[candidates]
    H --> J[plan_prune: referenced U newest-N excluded from remove]
    I --> J
    J -->|--apply| K[apply_prune: /proc-check then unlink, skip if open]
    J --> L[format_prune_report / prune_plan_to_json_dict]

    M[DIRECTORY] -->|scan, one walk incl. newest_mtime| N[candidates: threshold + fresh-enough]
    N --> O[plan_sweep]
    O -->|--apply, per --confirm NAME| P[apply_sweep: SLURM-only, walltime-aware, one at a time]
    P --> Q[_sweep_one: tar to temp, verify non-empty, atomic rename, rmtree]
    O --> R[format_sweep_report / sweep_plan_to_json_dict]

    S[SOURCE] -->|_measure_dir| T[plan_archive: size + file_count]
    T -->|--yes| U[apply_archive: sync_dir push, verify, write manifest, THEN rmtree]
    T --> V[format_archive_report / archive_plan_to_json_dict]
    U --> W[(archive-manifests/*.json)]
    W -->|plan_restore| X[apply_restore: sync_dir pull, optional delete_remote]

    Y[PATH...] -->|fclones group: size+hash| Z[duplicate groups]
    Z --> AA[format_duplicates_report / duplicates_to_json_dict]
    AA --> F

scan's walk and find-duplicates's hashing are both delegated to Rust CLIs for speed at multi-TB scale (see "System dependencies" above) instead of a hand-rolled os.walk/hashlib reimplementation. They are deliberately separate pipelines, not a shared one: scan must stay stat-only (safe to point at a 100%-full disk), and finding exact duplicates fundamentally requires reading bytes, which scan never does.

scitex_storage/
├── _scan.py        ← scan, scan_roots, ChildUsage, RootScan (fd-backed)
├── _duplicates.py  ← find_duplicates (fclones-backed)
├── _images.py      ← plan_prune, apply_prune, PruneCandidate, PrunePlan
├── _sweep.py       ← plan_sweep, apply_sweep, sweep_status, SweepPlan, SweepResult
├── _archive.py     ← plan_archive, apply_archive, plan_restore, apply_restore, ArchiveManifest
├── _report.py      ← format_report, format_duplicates_report, to_json_dict, format_prune_report, format_sweep_report, format_archive_report, format_size
└── _cli/           ← scan, find-duplicates, images prune, sweep, sweep-status, archive, restore, list-python-apis, mcp list-tools

1 Interfaces

scitex-storage scan [PATH ...] [--top N] [--sort size|files] [--max-depth D] [--json]
scitex-storage find-duplicates PATH [PATH ...] [--max-depth D] [--json]

scitex-storage images prune DIRECTORY [--keep N] [--pattern GLOB] [--apply] [--json]

$ scitex-storage images prune ~/.scitex/agent-container/containers/sac-base
scitex-storage images prune  /home/user/.scitex/agent-container/containers/sac-base
====================================================================================
1 referenced (always kept), 4 newest kept, 2 to remove

  WOULD REMOVE:
        1.3 GB  sac-base-2026-0512-212752.sif
        1.3 GB  sac-base-2026-0513-083535.sif

  2.6 GB reclaimable
  (dry-run — pass --apply to actually delete)

scitex-storage sweep DIRECTORY --threshold-files N [--min-age-hours H] [--apply --confirm NAME ...] [--min-remaining-seconds S] [--json]

$ scitex-storage sweep /data/gpfs/projects/punim0264/runs --threshold-files 5000
scitex-storage sweep  /data/gpfs/projects/punim0264/runs
==========================================================
threshold >= 5,000 files, min age 24h  (1 candidate(s), 0 skipped as too fresh)

  CANDIDATES (dry-run):
        18,204  old-run-42  (~18,203 inodes reclaimable)

  18,203 inodes reclaimable
  (dry-run — pass --apply --confirm NAME [--confirm NAME ...] to sweep)

scitex-storage sweep-status DIRECTORY [--json]

scitex-storage archive SOURCE --to nas|nas2 [--remote-path PATH] [--exclude PATTERN ...] [--checksum/--no-checksum] [--yes|-y] [--dry-run] [--json]
scitex-storage restore SOURCE [--delete-remote] [--yes|-y] [--dry-run] [--json]

import scitex_storage as ss

result = ss.scan("~/.scitex")           # read-only, stat-only walk
result.total_size                        # bytes across all children
result.total_files                       # inodes across all children
result.by_size()[0]                      # biggest child (ChildUsage)
result.by_file_count()[0]                # child with the most inodes

for r in ss.scan_roots(["~/.scitex", "~/proj"]):
    print(r.root, r.total_size, r.total_files)

plan = ss.plan_prune("~/.scitex/agent-container/containers/sac-base", keep=5)
plan.remove                              # candidates that would be deleted
plan.reclaimable_bytes                   # bytes those candidates hold

result = ss.apply_prune(plan)            # never touches referenced or open files
result.removed                           # candidates actually unlinked
result.skipped_in_use                    # candidates skipped: [(candidate, pids), ...]
result.reclaimed_bytes                   # bytes actually freed

sweep_plan = ss.plan_sweep("/data/gpfs/projects/punim0264/runs", threshold_files=5000)
sweep_plan.candidates                    # inode-hog children, old enough to be safe
sweep_plan.skipped_fresh                 # met the threshold but too recently touched

sweep_result = ss.apply_sweep(sweep_plan, confirm_names=["old-run-42"])  # SLURM-only
sweep_result.swept                       # [SweptCandidate(tar_path, member_count, ...), ...]
sweep_result.reclaimed_inodes

ss.sweep_status("/data/gpfs/projects/punim0264/runs")  # what's already swept

archive_plan = ss.plan_archive("~/proj/old-experiment", "nas2")
archive_plan.remote_path                 # default: mirrors the source path
manifest = ss.apply_archive(archive_plan)  # sync, verify, write manifest, remove source

restore_plan = ss.plan_restore("~/proj/old-experiment")  # reads the manifest back
ss.apply_restore(restore_plan)           # pulls it back; keeps the remote copy by default

# Separate, explicitly opt-in -- reads file contents to hash them.
groups = ss.find_duplicates(["~/proj/old-scan"])
for group in groups:
    print(len(group), "identical files:", group)

Roadmap (not implemented)

Discovery (scan), rotation (images prune, sweep), and migration (archive/restore) are layers 1-3 of a larger, safety-first design — scan → recommend → dry-run → copy → verify → quarantine → delete, never an immediate destructive action:

scitex-storage
├── Discovery       what exists (size + inodes)             <- scan (done)
├── Rotation        prune / tar superseded or hog files       <- images prune, sweep (done)
├── Classification  what kind of data it is                  <- planned
├── Policy          where it should live                     <- planned
├── Migration       safe copy/move (archive --to nas|nas2)     <- archive, restore (done)
├── Verification    checksum + count verify after move         <- rsync --checksum (done)
└── Retention       keep/quarantine/delete rules                 <- planned

Backends beyond ssh/rsync (S3-compatible, Gitea/GitHub) are planned but not present in this release.

Part of SciTeX

scitex-storage is part of SciTeX.

Four Freedoms for Research

0. The freedom to run your research anywhere — your machine, your terms.
1. The freedom to study how every step works — from raw data to final manuscript.
2. The freedom to redistribute your workflows, not just your papers.
3. The freedom to modify any module and share improvements with the community.

AGPL-3.0 — because we believe research infrastructure deserves the same freedoms as the software it runs on.

License

AGPL-3.0 — see LICENSE for details.