containerio 2.3.2

Azure Blob Storage handler with unified local/cloud interface

py-containerio

Shared Azure Blob Storage handler for internal and collaboration projects of Alliance SwissPass. Provides a unified interface for reading and writing data to both local filesystem and Azure Blob Storage, allowing seamless switching between local development and cloud environments.

Features

• Unified API: Same code works for local files and Azure Blob Storage
• Flexible configuration: Load Azure config from a dotenv file, from existing environment variables, or directly from strings (e.g. values from a TOML file)
• Polars integration: Native support for Polars DataFrames and LazyFrames
• Multiple auth methods: Device code flow (default), client credentials (CI/CD), or SAS tokens
• CLI tools: Unified io command for listing, downloading, uploading, deleting blobs and generating SAS tokens
• Memory-efficient: Streaming support with scan_* and sink_* methods

Install containerio

From PyPI

The package is available on PyPI: https://pypi.org/project/containerio/

pip install containerio

Or with uv:

uv add containerio
uv sync

Use containerio as a package dependency

From PyPI (recommended)

Add to your project's pyproject.toml:

[project]
dependencies = [
    "containerio",
]

Then run:

uv sync

From Git

For development or testing, install directly from the repository:

[project]
dependencies = [
    "containerio",
]

[tool.uv.sources]
containerio = { git = "https://codefloe.com/Alliance-SwissPass/py-containerio.git", tag = "v1.0.0" }

Usage in Other Projects

When installed as a dependency, containerio integrates seamlessly with your project:

• CLI tools are available - Run uv run io directly from your project
• Dotenv is the default convention - The CLI best-effort-loads .env.<name> matching --env <name> from your current working directory. Other formats are supported via a small wrapper script — see Providing config to the CLI.
• No configuration in containerio needed - Just add the dependency and configure in your project

Example workflow in your project:

# Your project directory
cd my-project

# Create your .env.staging file here
cp /path/to/.env.example .env.staging
# Edit .env.staging with your credentials

# List blobs in both containers
uv run io ls --env staging

# List blobs from the local filesystem (./data/ directory)
uv run io ls

# Generate SAS tokens (updates .env.staging)
uv run io generate-sas --env staging --update-env

Example Usage

The storage backend is chosen by what you pass into StorageHandler. Pass nothing (or config=None) for the local filesystem. Pass an AzureStorageConfig for cloud storage.

from containerio import AzureStorageConfig, StorageHandler

# Local filesystem (./data/ directory) — no config needed
handler = StorageHandler()

# Cloud storage — build an AzureStorageConfig explicitly, then pass it in.
# Option 1: read from os.environ (caller is responsible for populating it).
config = AzureStorageConfig.from_env()

# Option 2: load from a specific dotenv file (any path, relative or absolute).
config = AzureStorageConfig.from_env_file(path=".env.staging")
config = AzureStorageConfig.from_env_file(path="/etc/myapp/staging.env")

# Option 3: construct directly from strings (e.g. values loaded from a TOML file).
config = AzureStorageConfig(
    storage_account="<azure_storage_account>",
    tenant_id="<azure_tenant_id>",
)

# Open a handler against a named container. Permissions (read, write,
# delete) are enforced by Azure RBAC on your identity — the handler
# itself does not gate operations.
handler = StorageHandler(container="raw-data", config=config)

# Read files
df = handler.read_csv_blob("path/to/file.csv")
df = handler.read_excel_blob("path/to/file.xlsx")
df, metadata = handler.read_parquet_blob("path/to/file.parquet")

# Lazy scan (memory-efficient for large files)
lf, metadata = handler.scan_parquet_blob("large_file.parquet")
lf = handler.scan_csv_blob("large_file.csv")

# Write to a different container (re-use the same config)
curated = StorageHandler(container="curated", config=config)
curated.write_csv_blob("output.csv", df)
curated.write_parquet_blob("output.parquet", df)
curated.write_excel_blob("output.xlsx", df)

# Stream LazyFrame directly to storage
curated.sink_parquet_blob("output.parquet", lf)

# Upload any file (JSON, YAML, images, etc.)
curated.upload_blob("config.json", "./local/config.json")

# List, download, delete, move
blobs = handler.list_blobs()    # Returns List[BlobInfo]
handler.download_blob("file")   # Download to local ./data/
curated.delete_blob("file")     # Delete a blob or folder
curated.move_blob("old.csv", "archive/old.csv")  # Move/rename

Configuration

Environment Variables

Set STORAGE_ENV (or pass --env <name> to the CLI) to control the storage backend. The table below describes the CLI's best-effort dotenv preload — the library itself reads os.environ, and it is up to you how those vars get there (dotenv, TOML wrapper, direnv, exports, …).

Value	Behavior
local (default)	Uses local filesystem (./data/ directory)
prod	CLI preloads .env (if present), then reads os.environ
any other name	CLI preloads .env.<name> (if present), then reads os.environ — e.g. staging → .env.staging, dev → .env.dev, eu-prod → .env.eu-prod

Only local and prod are special. Any other value is treated as an environment name and the corresponding .env.<name> file is preloaded — add as many as you need.

Azure Configuration

For Azure storage, create a .env file (for prod) or .env.staging (for staging).

Important: Restrict permissions on your .env file so only you can access it:

chmod 600 .env

Example .env (production)

AZURE_STORAGE_ACCOUNT=<azure_storage_account>
AZURE_TENANT_ID=<azure_tenant_id>
AZURE_CLIENT_ID=<azure_client_id>

# For CI/CD with client-credentials auth
AZURE_CLIENT_ID_CONTAINER=<container_app_client_id>
AZURE_CLIENT_SECRET_CONTAINER=<container_app_client_secret>

# SAS tokens (optional, only if using SAS auth).
# Suffix is the container name, uppercased with dashes → underscores.
AZURE_STORAGE_SAS_TOKEN_RAW_DATA=<sas_token_for_raw_data>
AZURE_STORAGE_SAS_TOKEN_CURATED=<sas_token_for_curated>

Example .env.staging

Same shape as .env; the file's name is what makes it the staging config.

AZURE_STORAGE_ACCOUNT=<azure_storage_account>
AZURE_TENANT_ID=<azure_tenant_id>
AZURE_CLIENT_ID=<azure_client_id>
AZURE_CLIENT_ID_CONTAINER=<container_app_client_id>
AZURE_CLIENT_SECRET_CONTAINER=<container_app_client_secret>
AZURE_STORAGE_SAS_TOKEN_RAW_DATA_STG=<sas_token_for_staging_container>

Note: There is no registry of containers — you simply pass --container <name> (or container="..." in code) and the call is forwarded to Azure. Whether the container exists and what you can do with it is decided by Azure RBAC on your identity.

Authentication

Use io auth to manage authentication explicitly. Device-code login and status auto-discover the environment from your .env / .env.staging file, so --env is optional. SAS token and client credentials login require explicit --env:

# Device code login — auto-discovers env, verifies against a container
uv run io auth --env staging --container raw-data

# Check auth status across all methods
uv run io auth status

# Check status against a specific container (verifies methods end-to-end)
uv run io auth status --env staging --container raw-data

# SAS token login — requires --env and --container
uv run io auth login --method sas-token --env staging --container raw-data

# Client credentials — for CI/CD pipelines
uv run io auth login --method client-credentials --env prod --container raw-data

# Clear cached device code tokens
uv run io auth clear

# Force re-authentication (clears cache first)
uv run io auth login --env staging --container raw-data --force

The chosen auth method is saved to the session. Subsequent commands (ls, download, etc.) automatically use it.

Authentication Methods

1. Device Code Flow (recommended for development)

   • Interactive browser-based authentication
   • Tokens are cached locally for 90 days
   • No credentials stored in files

2. Client Credentials

   • For CI/CD pipelines
   • Use io auth login --method client-credentials --env <env> --container <name> to authenticate
   • Requires (in the env file for that environment):
     • AZURE_TENANT_ID: Service principal's tenant ID
     • AZURE_CLIENT_ID_CONTAINER: Service principal's client ID
     • AZURE_CLIENT_SECRET_CONTAINER: Service principal's client secret

3. SAS Token

   • Use pre-generated tokens with limited validity
   • Good for sharing temporary access

CLI Tools

The package provides a unified io command with subcommands for all storage operations. All commands accept --env prod|staging (selects the env file) and --container <name> (selects the Azure container).

Command	Usage	Description
auth	uv run io auth [login|status|clear] [--method M] [--force]	Manage authentication
ls	uv run io ls --env <env> --container <name>	List blobs in a container
download	uv run io download <blob_name> --env <env> --container <name>	Download a blob to local data/ directory
upload	uv run io upload <blob_name> <file_path> --env <env> --container <name>	Upload a local file to a container
rm	uv run io rm <path> --env <env> --container <name>	Delete a blob or folder from a container
mv	uv run io mv <source> <dest> [--overwrite] --env <env> --container <name>	Move (rename) a blob within a container
generate-sas	uv run io generate-sas --env <env> --container <name> [--days N] [--update-env] [--write]	Generate an Azure Storage SAS token for a container (read+list by default; add --write for write+delete)

Examples

# Authenticate with device code flow against a container
uv run io auth --env staging --container raw-data

# Check authentication status
uv run io auth status

# List blobs from the local filesystem (./data/) — no --env / --container needed
uv run io ls

# List blobs in an Azure container
uv run io ls --env staging --container raw-data

# Download a blob to the local data/ directory
uv run io download path/to/file.csv --env staging --container raw-data

# Upload a local file
uv run io upload path/to/blob.csv ./local/file.csv --env staging --container curated

# Delete a blob or folder
uv run io rm path/to/blob.csv --env staging --container curated
uv run io rm old-folder/ --env staging --container curated

# Move a blob within a container
uv run io mv data/old.csv archive/old.csv --env staging --container curated

# Overwrite an existing destination
uv run io mv data/old.csv data/existing.csv --env staging --container curated --overwrite

# Generate a read+list SAS token for a container and update the .env.staging file
uv run io generate-sas --env staging --container raw-data --update-env

# Generate a read+write+delete+list SAS token (opt-in via --write)
uv run io generate-sas --env staging --container curated --update-env --write

# Print a SAS token for a prod container valid for 3 days
uv run io generate-sas --env prod --container raw-data --days 3

Session

When doing multiple operations against the same container, use session to save defaults so you don't have to repeat --env and --container on every command. Both --env and --container are required to start a session.

# Start a session — saves env and container
uv run io session --env staging --container raw-data

# Show active session
uv run io session

# Now these use the saved defaults
uv run io ls
uv run io rm old-folder/
uv run io mv data/old.csv archive/old.csv

# Explicit flags still override the session
uv run io ls --container curated

# End the session
uv run io session clear

The session is stored in .containerio/session.json in the current directory. It is project-local and git-ignored.

The session is also available programmatically via the Session class:

from containerio.session import Session

session = Session()
print(session.auth_method)  # e.g. 'client-credentials'
print(session.env)          # e.g. 'staging'
print(session.container)    # e.g. 'raw-data'
print(session.is_active)    # True if any session data exists

CLI options:

• --env <name>: selects the environment. The CLI best-effort-preloads .env.<name> (no-op if missing) and uses the name as a label in prints, sessions, and SAS file write-back. Any name works (prod, staging, dev, eu-prod, …); local means the local filesystem.
• --container: the Azure container name to target
• --days: SAS token validity in days, max 7 (default: 1, generate-sas only)
• --update-env: write SAS token to env file instead of printing (generate-sas only)
• --write: opt in to write+delete permissions (generate-sas only). Default is read+list.

Note: SAS tokens are generated using a user delegation key which has a maximum validity of 7 days (Azure limitation). See Microsoft documentation for details.

The token's effective permissions are the intersection of the permissions requested here and the Azure RBAC permissions held by your identity (the user-delegation-key requester) at generation time — so requesting --write only produces a write-capable token if your RBAC role allows it. See Microsoft documentation on user delegation SAS for details.

API Reference

StorageHandler

Main class for storage operations.

StorageHandler(
    container: Optional[str] = None,
    config: Optional[AzureStorageConfig] = None,
    auth_type: Optional[str] = None,
    session: Optional[Session] = None,
)

• container: The container name (required for cloud storage, ignored for the local filesystem). Whether the container exists and what you can do with it is decided by Azure RBAC on your identity.
• config=None: Local filesystem mode (uses ./data/ directory).
• config=<AzureStorageConfig>: Cloud storage mode. Build via AzureStorageConfig(...) or AzureStorageConfig.from_env_file(path=...).
• auth_type=None: Falls back to session auth method, then device code flow
• auth_type="device-code": Device code flow (interactive). Uses config.device_code_client_id.
• auth_type="sas-token": SAS token authentication (reads AZURE_STORAGE_SAS_TOKEN_<UPPER_CONTAINER_NAME>)
• auth_type="client-credentials": Client credentials (service principal). Uses config.client_id and config.client_secret.
• session=None: Creates a new Session (skipped for the local filesystem). Pass an existing Session instance for dependency injection.

Read Methods

Method	Description
read_parquet_blob(blob_name)	Read parquet file, returns (DataFrame, metadata)
read_csv_blob(blob_name, **kwargs)	Read CSV file into DataFrame
read_excel_blob(blob_name, **kwargs)	Read Excel file into DataFrame
read_json_blob(blob_name, **kwargs)	Read JSON file into DataFrame
scan_parquet_blob(blob_name)	Lazy scan parquet, returns (LazyFrame, metadata)
scan_csv_blob(blob_name, **kwargs)	Lazy scan CSV file

Write Methods

Method	Description
write_parquet_blob(blob_name, df)	Write DataFrame to parquet
write_csv_blob(blob_name, df, **kwargs)	Write DataFrame to CSV
write_excel_blob(blob_name, df, **kwargs)	Write DataFrame to Excel
write_json_blob(blob_name, df, **kwargs)	Write DataFrame to JSON
sink_parquet_blob(blob_name, lf)	Stream LazyFrame to parquet

Utility Methods

Method	Description
upload_blob(blob_name, file_path, progress_hook)	Upload a local file to storage
delete_blob(blob_name)	Delete a blob, file, or folder recursively. Returns List[str] of deleted paths
move_blob(source, destination, overwrite)	Move (rename) a blob within the same container
download_blob(blob_name, progress_hook)	Download blob to local data/ directory. Returns Optional[Path]
list_blobs()	List all blobs, returns List[BlobInfo]

AzureStorageConfig

Configuration dataclass for Azure storage settings. Fields: storage_account, tenant_id (optional), dotenv_path (optional — the file the config was loaded from; useful in logs and repr), client_id (optional), client_secret (optional) and device_code_client_id (optional). storage_endpoint is a derived property — https://<storage_account>.blob.core.windows.net.

The config is the single carrier of account and identity data: the handler reads these fields and never reads environment variables for them itself. from_env() is the one seam that maps env vars onto the config (AZURE_CLIENT_ID_CONTAINER → client_id, AZURE_CLIENT_SECRET_CONTAINER → client_secret, AZURE_CLIENT_ID → device_code_client_id). The lone exception is the SAS token, which is container-scoped and therefore resolved by the handler from AZURE_STORAGE_SAS_TOKEN_<NAME> at use time.

client_id and client_secret apply to client-credentials auth; device_code_client_id applies to device-code auth. Setting client_id/client_secret per config lets you drive multiple containers backed by different service principals in the same process:

raw = AzureStorageConfig(
    storage_account="myaccount",
    tenant_id="<tenant>",
    client_id="<raw_sp_client_id>",
    client_secret="<raw_sp_secret>",
)
curated = AzureStorageConfig(
    storage_account="myaccount",
    tenant_id="<tenant>",
    client_id="<curated_sp_client_id>",
    client_secret="<curated_sp_secret>",
)

raw_handler = StorageHandler(container="raw-data", config=raw, auth_type="client-credentials")
curated_handler = StorageHandler(container="curated", config=curated, auth_type="client-credentials")

The env name ("staging", "prod", …) is intentionally not a field on the config. That concept lives at the CLI and Session layer — see --env, Session.env, STORAGE_ENV. The config carries data, not an environment label.

Three ways to build a config:

from containerio import AzureStorageConfig

# 1. Read from os.environ (no file involved).
#    Vars must already be set (dotenv preload, direnv, exports, etc.).
config = AzureStorageConfig.from_env()

# 2. Load a specific dotenv file (any path, relative or absolute).
config = AzureStorageConfig.from_env_file(path=".env.staging")
config = AzureStorageConfig.from_env_file(path="/etc/myapp/staging.env")

# 3. Construct directly from strings (e.g. values loaded from a TOML file).
config = AzureStorageConfig(
    storage_account="myaccount",
    tenant_id="<tenant>",
)

# Access configuration
print(config.storage_account)   # "myaccount"
print(config.storage_endpoint)  # "https://myaccount.blob.core.windows.net"
print(config.dotenv_path)       # PosixPath('.env.staging') for option 2; None otherwise

Errors:

• from_env() raises ValueError if AZURE_STORAGE_ACCOUNT is not in os.environ.
• from_env_file(path) raises FileNotFoundError if the path does not exist, then ValueError if AZURE_STORAGE_ACCOUNT is still missing after loading.

Providing config to the CLI

The CLI reads required Azure variables from os.environ — it does not parse any specific config format itself. This keeps the CLI format-agnostic and lets each project choose how to manage config.

Dotenv is the default. If a .env.<name> file matching --env <name> exists, the CLI auto-loads it via python-dotenv before reading os.environ. No extra setup — drop a .env.staging next to your code and io ls --env staging works.

If you don't use dotenv, you populate os.environ yourself before invoking the CLI. Any of these work:

• Wrapper script that reads your config format and exports vars (see TOML example below).
• direnv that exports vars automatically on cd.
• CI / container injection — the platform sets vars in the environment before the CLI runs.
• Plain shell exports: export AZURE_STORAGE_ACCOUNT=... && io ls --env staging --container raw-data.

The CLI flow is the same in every case:

1. main() tries load_dotenv(".env.<name>") — silently no-ops if the file isn't there.
2. _build_config() calls AzureStorageConfig.from_env(), which reads whatever is now in os.environ.
3. If no vars are set anywhere, from_env() raises a clear AZURE_STORAGE_ACCOUNT not set error.

Example: TOML config via a wrapper script

Your project has a containerio.toml (or any name) with per-env sections:

[staging]
storage_account = "myacct-staging"
tenant_id = "tid-staging"

[prod]
storage_account = "myacct-prod"
tenant_id = "tid-prod"

Add a small wrapper that reads the section selected by --env and populates os.environ, then hands off to the CLI:

# scripts/io.py
import argparse
import os
import tomllib

parser = argparse.ArgumentParser(add_help=False)
parser.add_argument("--env", default=None)
args, _ = parser.parse_known_args()

if args.env and args.env != "local":
    with open("containerio.toml", "rb") as toml_file:
        section = tomllib.load(toml_file)[args.env]
    os.environ["AZURE_STORAGE_ACCOUNT"] = section["storage_account"]
    os.environ["AZURE_TENANT_ID"] = section["tenant_id"]

from containerio.tools.cli import main

main()

Invoke it the same way you would the bare CLI:

# read containerio.toml's [staging] section, then run ls
uv run python scripts/io.py ls --env staging --container raw-data

The wrapper is the only TOML-aware piece. containerio itself stays format-agnostic. The same pattern works for YAML, JSON, Vault lookups, or anything else — read your source, set os.environ, call main().

Environment helpers

Small helpers for working with environment names.

from containerio.azure_config import env_file, is_local

is_local("local")      # True
is_local("prod")       # False
is_local(None)         # True (defaults to local)

env_file("prod")       # ".env"
env_file("staging")    # ".env.staging"
env_file("dev")        # ".env.dev"
env_file("local")      # raises ValueError

Development

# Clone and install
git clone ssh://[e-mail]/Alliance-SwissPass/py-containerio.git
cd py-containerio
uv sync --group dev

# Run tests
uv run pytest

# Lint and format
uv run ruff check
uv run ruff format

Version Management with git-sv

This project uses git-sv for semantic versioning and changelog generation.

The .gitsv/config.yaml file defines how versions are bumped based on commit messages.

1. Commit messages: Follow conventional commits format
2. Version bump: The CI pipeline automatically runs git-sv bump based on commit history
3. Changelog: The CI pipeline generates the changelog using git-sv changelog and pins it as a Git issue
4. Release: The CI pipeline automatically handles tagging and publishing

The CI pipeline will automatically:

• Bump the version based on commit history
• Generate and pin the changelog as a Git issue
• Build and publish the package to Forgejo
• Create a release with the changelog notes

Publishing

Via CI Pipeline (recommended)

The repository includes a Crow CI pipeline that automatically builds, tests, and publishes the package when a git tag is pushed:

1. Merge all branches you want to have part of a new release into main
2. git sv rn: check tag for upcoming semantic release version and associated changelog.
3. If not already added, add your private key (whose public key is associated with your Codefloe Git user) to the SSH agent. ssh-add ~/.ssh/<your-private-key>
4. git sv tag: generate tag with version and push it to remote in one go, based on git commit parsing.

The pipeline will:

• Run linting and tests
• Update the version in pyproject.toml automatically
• Generate changelog from git history using git-sv
• Build and publish the package to Forgejo and PyPI
• Create a release with changelog notes
• Push version updates back to the repository

Note: For CI publishing, ensure the UV_PUBLISH_TOKEN secret is configured in your Crow CI pipeline settings.

Security Best Practices

1. Never commit credentials: Always add .env* to your .gitignore file
2. Restrict file permissions: Use chmod 600 .env to limit access to your credentials
3. Use short-lived tokens: SAS tokens have a maximum validity of 7 days
4. Rotate secrets regularly: Especially for production environments and CI/CD pipelines
5. Use device code flow: For development to avoid storing credentials in files
6. Environment isolation: Use separate credentials for staging and production
7. Limit token scope: When generating SAS tokens, specify the minimum required permissions

References

• Delegate access with shared access signature
• Create user delegation SAS
• Get user delegation key

License

Alliance SwissPass https://allianceswisspass.ch