file-watchman 0.1.0

Work directory manifest and delta library

file-watchman

A Python library that scans work directories, produces JSON manifests, and computes file-level deltas for incremental uploads and workflow monitoring.

Features

• Directory scanning — walk a directory tree with configurable include/exclude glob rules
• Manifest snapshots — deterministic, sorted JSON manifests with file metadata
• Delta computation — detect new, changed, and deleted files between snapshots
• Category-driven hashing — SHA-256 for critical files (checkpoints), skip for large files (trajectories)
• Atomic writes — safe manifest persistence via tmp + fsync + rename
• Zero runtime dependencies — stdlib only, Python 3.10+

Installation

From GitHub:

pip install git+https://github.com/ATTMOS/file-watchman.git

For development:

git clone https://github.com/ATTMOS/file-watchman.git
cd file-watchman
pip install -e ".[dev]"

Quick Start

from file_watchman import (
    ScanRules,
    HashPolicy,
    scan_manifest,
    diff_manifests,
    load_manifest,
    write_manifest_atomic,
)

# Configure scan rules
rules = ScanRules(
    includes=["**/*"],
    excludes=[".git/**", "**/__pycache__/**", "**/*.tmp"],
    max_files=50000,
)

# Configure hashing policy
hash_policy = HashPolicy(
    required_categories={"checkpoint", "restart"},
    optional_categories={"log"},
    never_categories={"trajectory"},
    max_hash_bytes=512 * 1024 * 1024,
)

# Scan a directory
manifest_new = scan_manifest(
    root="/scratch/job_123",
    rules=rules,
    hash_policy=hash_policy,
    job_id="job-123",
)

# Compare with a previous manifest
manifest_prev = load_manifest("/scratch/job_123/manifest_prev.json")
delta = diff_manifests(manifest_prev, manifest_new)

print(f"Uploads: {len(delta.uploads)}, Deletions: {len(delta.deletions)}")

# Persist the new manifest atomically
write_manifest_atomic("/scratch/job_123/manifest_new.json", manifest_new)

Custom Categorization

Assign categories to files using glob-based rules:

from file_watchman.rules import make_categorizer

categorize = make_categorizer([
    ("**/cpt_*.pkl", "checkpoint"),
    ("**/*.rst7", "restart"),
    ("**/*.log", "log"),
    ("**/*.nc", "trajectory"),
])

manifest = scan_manifest(
    root="/scratch/job_123",
    rules=rules,
    hash_policy=hash_policy,
    categorize=categorize,
)

Files not matching any rule default to the "other" category.

API Reference

See docs/api.md for the complete API reference.

License

MIT