cloud-vfs

Manual cloud blob virtual filesystem for repos with large artifacts. Keep primary disks small: data lives in Azure Blob or S3, local paths keep tiny inline refs (same path) or .cloudstub directory pointers, and a machine-maintained per-file inventory tracks explicit cloud paths.

Design: generic source (cloud archive) and target (filesystem) — see docs/DESIGN.md.

Works with Cursor / Claude agents or plain shell + Azure CLI / AWS CLI.

Why cloud-vfs (not DVC / Git LFS)

Best for: disk hygiene + lazy fetch + explicit offload when projects store large files under data/ (or policy-defined prefixes).

Features

• Per-file inventory — .cloud-vfs/index/<shard>.json with local, blob, sha256, etag, state
• Lazy fetch — cloud-vfs ensure <path> (single file or whole tree)
• Manual offload — hash before delete; verifies the remote object (exists + size) before removing local; --dry-run first
• Drift audit — cloud-vfs reconcile compares disk ↔ inventory ↔ blob (HEADs each remote blob by default; --no-verify-blobs to skip)
• Large-data scope — default ≥ 50 MB under data/; prefix overrides for weights, etc.
• Multi-cloud — Azure Blob and AWS S3
• Cursor skill — cloud-vfs init --skill

No auto-tracking, no cron, no background jobs.

Install

pip install cloud-vfs

Or from GitHub:

pip install git+https://github.com/sahasrarjn/cloud-vfs.git
curl -fsSL https://raw.githubusercontent.com/sahasrarjn/cloud-vfs/main/install.sh | bash

Requires Python 3.9+, az and/or aws CLI, cloud credentials, and azcopy v10 (recommended for multi-GB blob transfers).

Try it in 5 minutes

pip install cloud-vfs
cloud-vfs try
cd cloud-vfs-try
cp .cloud-vfs/config.env.example .cloud-vfs/config.env   # set a TEST bucket
cloud-vfs doctor --roundtrip
./scripts/create-sample.sh
cloud-vfs offload --dry-run data/sample && cloud-vfs offload data/sample
cloud-vfs ensure data/sample

Full walkthrough: docs/TRY.md. Same demo lives in examples/minimal-demo/ if you cloned this repo.

Quick start (your project)

Point at any repo or folder (must be writable; run from repo root or pass --path):

cd /path/to/your-ml-repo
cloud-vfs init --path . --skill
cp .cloud-vfs/config.env.example .cloud-vfs/config.env   # set bucket (see config.env.example)
cloud-vfs doctor --roundtrip

cloud-vfs scan                    # what large files can you offload?
cloud-vfs scan --add              # add them to manifest (no upload yet)
cloud-vfs offload --dry-run       # preview: sizes + cloud target
cloud-vfs offload data/your_run   # upload + stub (you choose paths)
cloud-vfs ensure data/your_run    # fetch back when needed

Optional: cloud-vfs register <path> indexes sha256 without upload; cloud-vfs status --drift audits inventory.

Two layers

Inventory rows are created by offload, register, and reconcile --fix-index — never hand-edited.

Commands

Project layout

your-project/
  .cloud-vfs/
    config.env              # account names (commit)
    secrets.env             # keys (gitignored)
    manifest.json           # folder-level policy (commit)
    inventory-policy.json   # min size, include/exclude (commit)
    index/                  # per-file inventory shards
      data/
        ADME.json             # commit benchmark shards
        generated/            # often gitignored — regenerate from blob
  data/
    big.npy                   # inline JSON ref when single file offloaded
    big/.cloudstub            # directory pointer when tree offloaded
  .cursor/skills/cloud-vfs/   # optional

Tracking scope (defaults)

See docs/INVENTORY.md.

One or two archives (Azure and/or AWS)

Set LOCAL_PROVIDER=azure or aws in .cloud-vfs/config.env.

Azure: AZ_LOCAL_*, AZ_REMOTE_* + keys in secrets.env

AWS: AWS_LOCAL_BUCKET, AWS_LOCAL_REGION (uses aws CLI credentials)

Manifest archive keys: local_archive (primary), remote_staging (secondary). See docs/SOURCE_TARGET.md.

Agents

cloud-vfs ensure path/to/file          # before reading cloud-only paths
cloud-vfs register path/to/new.npy     # after creating outputs ≥ min size
cloud-vfs reconcile                    # after compute runs
cloud-vfs offload --dry-run path       # always dry-run + confirm with user
cloud-vfs offload path

Never hand-edit .cloud-vfs/index/*.json.

Environment variables

Update notice

cloud-vfs prints a one-line, non-blocking reminder to stderr when a newer release is on PyPI (checked at most once a day, cached under ~/.cache/cloud-vfs/). It never installs anything. It stays silent in CI, when output is piped, and when CLOUD_VFS_NO_UPDATE_CHECK=1 is set.

Documentation

• docs/CLOUD_VFS.md — workflow, stubs, drift
• docs/INVENTORY.md — policy, shards, git hygiene
• docs/AGENTS.md — rules for coding agents
• docs/ROBUSTNESS.md — verify, guard, orphan blobs, prod vs cloud-vfs bucket
• docs/YOUR_REPO.md — scan and offload in your existing repo
• docs/TEAM.md — multi-contributor playbook: commit vs blob vs inventory, clone workflow
• docs/TRY.md — 5-minute try guide
• examples/minimal-demo/ — demo sources (also bundled in cloud-vfs try)
• docs/PUBLISHING.md — PyPI release process

License

MIT — see LICENSE.