HuggingFaceStorage

Python library for cloud-style file storage backed by a private Hugging Face dataset repository.

Features

• Immutable file version history per logical remote path
• Soft delete via tombstone versions
• Content-addressed blob storage (sha256) to avoid duplicate uploads
• HF_TOKEN-based authentication
• Public API: put, put_zip, get, list, delete, history

Project Structure

HuggingFaceStorage/
  src/hf_storage/
  tests/unit/
  tests/integration/
  requirements.txt
  pyproject.toml

Setup

1. Create the virtual environment (Python 3.11):

py -3.11 -m venv .venv

2. Activate:

.\.venv\Scripts\Activate.ps1

3. Install dependencies:

python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip install -e .

4. Deactivate when finished:

deactivate

Authentication

Set your Hugging Face token before using the library:

$env:HF_TOKEN = "hf_xxx"

Quick Example

from hf_storage import HFStorage, StorageConfig

storage = HFStorage(StorageConfig(repo_id="your-namespace/your-private-dataset"))
storage.setup(create_if_missing=True, private=True)

version = storage.put("local.txt", "docs/local.txt")
zip_version = storage.put_zip("my_folder", "archives/my_folder")
storage.get("docs/local.txt", "restored.txt")
entries = storage.list(prefix="docs/")
history = storage.history("docs/local.txt")
deleted = storage.delete("docs/local.txt")

Running Tests

Unit tests:

pytest tests/unit

Integration tests (real HF repo):

$env:HF_STORAGE_INTEGRATION = "1"
$env:HF_STORAGE_TEST_REPO = "your-namespace/your-private-dataset"
pytest tests/integration

Publishing (Maintainers)

publish.bat is maintainer tooling for package release workflow only. It is not part of the public runtime API.

Set Twine credentials via environment variables:

$env:TWINE_USERNAME = "__token__"
$env:TWINE_TEST_PASSWORD = "pypi-<testpypi-token>"
$env:TWINE_PASSWORD = "pypi-<pypi-production-token>"

Default release target is TestPyPI:

.\publish.bat

Publish to production PyPI explicitly:

.\publish.bat pypi

What publish.bat does:

• Runs unit tests (pytest tests/unit)
• Builds wheel + sdist (python -m build)
• Validates artifacts (twine check dist/*)
• Uploads to TestPyPI by default, or PyPI when pypi is passed

Credential behavior:

• .\publish.bat (default TestPyPI) uses TWINE_TEST_PASSWORD
• .\publish.bat pypi (production) uses TWINE_PASSWORD
• TWINE_USERNAME must be __token__ for both

Important: bump package version before each release. PyPI/TestPyPI do not allow re-uploading the same version.

One-command Zip Upload (Windows)

Use the batch wrapper to zip and upload a file or directory:

.\put_zip.bat "C:\path\to\folder_or_file" "backups/my_archive"

Notes:

• HF_TOKEN and HF_STORAGE_REPO_ID are read from .env (or current env vars).
• If the remote path does not end with .zip, .zip is appended automatically.

List and Download (Windows)

List stored logical paths:

.\list_files.bat

Include soft-deleted entries too:

.\list_files.bat 1

Download latest version by logical path:

.\get_file.bat "backup/venv.zip" ".\downloads\venv.zip"

Download a specific version:

.\get_file.bat "backup/venv.zip" ".\downloads\venv.zip" "version_id_here"

Soft delete a logical path:

.\delete_file.bat "backup/venv.zip"

Hard delete a logical path (removes manifest entry and unreferenced blob objects):

.\delete_file.bat "backup/venv.zip" hard