azure-blob-tui

Terminal TUI for browsing Azure Blob Storage, plus Python helpers to read/write blobs directly from your training code (no local files required).

First-time setup

Run the TUI once to configure account/container/prefix and (optionally) store SAS in an encrypted local file. On later runs, if SAS is stored, you only need the passphrase (no need to re-enter SAS).

azure-blob-tui --configure

During first run you will be prompted for:

• account name / container name / default prefix
• whether to store SAS in an encrypted local file
• (if yes) the SAS token and a passphrase to encrypt it

Later runs:

• If SAS is stored, you will only be prompted for the passphrase.
• If you set AZURE_BLOB_TUI_PASSPHRASE, no prompt is needed.

Use the TUI

azure-blob-tui

Reconfigure

azure-blob-tui --configure

Python API (no local files)

All helpers use the same config (account/container/default prefix) and SAS token stored by azure-blob-tui --configure. If AZURE_BLOB_TUI_PASSPHRASE is set, no prompt is needed.

Default prefix behavior:

• Default prefix is not applied automatically.
• Pass use_default_prefix=True to opt in per call.

blob_open (file-like stream)

import torch
from azure_blob import blob_open

# Save directly to Blob
with blob_open("checkpoints/step-100/model.pt", "wb") as f:
    torch.save(model.state_dict(), f)

# Load directly from Blob
with blob_open("checkpoints/step-100/model.pt", "rb") as f:
    state = torch.load(f, weights_only=False)

Save JSON/YAML/text to Blob

import io
import json
from azure_blob import blob_open

with blob_open("artifacts/config.json", "wb") as raw:
    with io.TextIOWrapper(raw, encoding="utf-8") as f:
        json.dump({"lr": 1e-4}, f)

blob_url (signed URL helper)

from azure_blob import blob_url

url = blob_url("images/cat.png")

Storage helpers

from azure_blob import (
    clear_cache,
    download_dir,
    download_file,
    list_blobs,
    prefetch_blobs,
    read_blob_bytes,
    upload_dir,
    upload_file,
)

upload_file("local.txt", "artifacts/local.txt")
download_file("artifacts/local.txt", "local_copy.txt", backend="auto", use_cache=False)

# Preload to local cache (faster repeated reads)
prefetch_blobs(["artifacts/local.txt", "artifacts/another.txt"])

# Reused metadata: cache on demand
meta = read_blob_bytes("artifacts/index.jsonl", use_cache=True)

# Clear cached objects (single blob or whole prefix)
clear_cache(blob_name="artifacts/local.txt")
clear_cache(prefix="artifacts/")

for name in list_blobs(prefix="artifacts/"):
    print(name)

BlobContext (high-level workflow helper)

BlobContext wraps a configured ContainerClient and adds convenience helpers for prefix resolution, listing, and in-memory read/write.

import os
from azure_blob import BlobContext

os.environ["AZURE_BLOB_TUI_PASSPHRASE"] = "your-passphrase"

ctx = BlobContext.from_config()

for prefix in ctx.iter_prefixes(prefix="overview/among/"):
    print(prefix)

data = ctx.read_bytes("overview/among/123/topdown.png")
# Cache only when repeated reads are expected (for example JSONL metadata)
meta = ctx.read_bytes("dataset/index.jsonl", use_cache=True)
# For strict freshness, bypass cache:
latest = ctx.read_bytes(
    "overview/among/123/topdown.png",
    backend="auto",
    force_refresh=True,
)
ctx.write_bytes(
    "overview/among/123/processed.png",
    data,
    content_type="image/png",
)

Fast Read Mode

Read-heavy APIs support backend:

• backend="auto" (default): use AzCopy for larger/batch reads when available, otherwise SDK.
• backend="sdk": force Azure Python SDK path.
• backend="azcopy": force AzCopy path (requires AzCopy + SAS).

backend="auto" selects AzCopy only when AzCopy is installed and a SAS token is available; otherwise it falls back to SDK automatically.

Optional environment variables:

• AZURE_BLOB_TUI_READ_BACKEND=auto|sdk|azcopy
• AZURE_BLOB_TUI_READ_WORKERS=16 (directory and prefetch parallelism)
• AZURE_BLOB_TUI_MAX_CONCURRENCY=8 (per-blob SDK transfer concurrency)
• AZURE_BLOB_TUI_AZCOPY_PATH=/path/to/azcopy

Install AzCopy (optional, recommended for faster bulk download):

azcopy --version

Cache Behavior

Read APIs are non-cached by default (use_cache=False).

• Cache default location: ~/.cache/azure-blob-tui/blob-cache
• Override with: AZURE_BLOB_TUI_CACHE_DIR=/your/cache/dir
• Enable globally with: AZURE_BLOB_TUI_CACHE_ENABLED=1
• Cache hard limit (default 10GB): AZURE_BLOB_TUI_CACHE_MAX_BYTES=10GB
• Force bypass cache per call with: force_refresh=True

If files are updated by other writers, either:

• call read APIs with force_refresh=True, or
• clear stale entries via clear_cache(...).

When cache exceeds the configured hard limit, the oldest cached files are deleted automatically.

AZURE_BLOB_TUI_CACHE_MAX_BYTES accepts raw bytes or unit suffixes (KB, MB, GB), for example: 10737418240, 10240MB, 10GB.

Common Parameter Choices

• Realtime image fetching (use once): download_file(..., use_cache=False, backend="auto")
• Reused JSON/JSONL metadata: read_blob_bytes(..., use_cache=True) or ctx.read_bytes(..., use_cache=True)
• Strong freshness (ignore local cache): add force_refresh=True
• Batch folder pull: download_dir(..., workers=16, backend="auto", use_cache=False) for one-shot jobs

download_file(local_path=...) always writes the requested blob into your target local path. use_cache only controls whether an extra cache copy is kept.

Recommended Pattern (JSONL + Images)

from azure_blob import read_blob_bytes, download_file
import json

# 1) Metadata is reused -> cache it
records = [
    json.loads(line)
    for line in read_blob_bytes("dataset/index.jsonl", use_cache=True)
    .decode("utf-8")
    .splitlines()
]

# 2) Images are one-shot -> do not cache
for rec in records:
    blob_name = rec["image_blob"]
    local_path = f"/tmp/images/{rec['id']}.jpg"
    download_file(blob_name, local_path, use_cache=False, backend="auto")
    # use image immediately ...

Choosing the right helper

• BlobContext: best for workflows that need listing, grouping, and in-memory read/write without touching disk.
• blob_open: best for file-like streaming APIs (e.g., torch.save/load, TextIOWrapper, or very large files).
• read_bytes/write_bytes: best for small-to-medium blobs where a full in-memory buffer is fine (images, JSON, etc.).
• list_blobs/iter_prefixes: best for enumeration and directory-like traversal.
• download_*/upload_*: local file paths (avoid these if you want no local storage).
• prefetch_blobs: best for pre-warming cache before training/evaluation loops.

Notes

• SAS tokens are not stored in the config file, but in an encrypted local file.
• blob_open works for any file type (torch checkpoints, JSON, YAML, text, etc.).