containerio 1.0.0

Azure Blob Storage handler with unified local/cloud interface

py-containerio

Shared Azure Blob Storage handler for internal and collaboration projects of Alliance SwissPass. Provides a unified interface for reading and writing data to both local filesystem and Azure Blob Storage, allowing seamless switching between local development and cloud environments.

Features

• Unified API: Same code works for local files and Azure Blob Storage
• Environment-based switching: Use STORAGE_ENV to switch between local/staging/prod
• Polars integration: Native support for Polars DataFrames and LazyFrames
• Multiple auth methods: Device code flow (default), client credentials (CI/CD), or SAS tokens
• CLI tools: Unified io command for listing, downloading, uploading, deleting blobs and generating SAS tokens
• Memory-efficient: Streaming support with scan_* and sink_* methods

Install containerio

From PyPI

The package is available on PyPI: https://pypi.org/project/containerio/

pip install containerio

Or with uv:

uv add containerio
uv sync

Use containerio as a package dependency

From PyPI (recommended)

Add to your project's pyproject.toml:

[project]
dependencies = [
    "containerio",
]

Then run:

uv sync

From Forgejo Package Registry

If you need the Forgejo version, configure the custom index:

[project]
dependencies = [
    "containerio",
]

[tool.uv.sources]
containerio = { index = "forgejo" }

[[tool.uv.index]]
name = "forgejo"
url = "https://codefloe.com/api/packages/Alliance-SwissPass/pypi/simple"

Then run:

uv sync

From Git

For development or testing, install directly from the repository:

[project]
dependencies = [
    "containerio",
]

[tool.uv.sources]
containerio = { git = "https://codefloe.com/Alliance-SwissPass/py-containerio.git", tag = "v1.0.0" }

Usage in Other Projects

When installed as a dependency, containerio integrates seamlessly with your project:

• CLI tools are available - Run uv run io directly from your project
• .env files live in your project - The storage module reads .env and .env.staging from your current working directory, not from the containerio package
• No configuration in containerio needed - Just add the dependency and configure in your project

Example workflow in your project:

# Your project directory
cd my-project

# Create your .env.staging file here
cp /path/to/.env.example .env.staging
# Edit .env.staging with your credentials

# List blobs in both containers
uv run io ls --env staging

# List blobs in local mode (./data/ directory)
uv run io ls

# Generate SAS tokens (updates .env.staging)
uv run io generate-sas --env staging --update-env

Example Usage

import os
from containerio import StorageHandler

# Set environment (defaults to "local" which uses ./data/ directory)
os.environ["STORAGE_ENV"] = "staging"
# or "prod"

# Read operations (uses read container)
handler = StorageHandler(container="ro")

# Read files
df = handler.read_csv_blob("path/to/file.csv")
df = handler.read_excel_blob("path/to/file.xlsx")
df, metadata = handler.read_parquet_blob("path/to/file.parquet")

# Lazy scan (memory-efficient for large files)
lf, metadata = handler.scan_parquet_blob("large_file.parquet")
lf = handler.scan_csv_blob("large_file.csv")

# Write operations (uses write container)
handler_rw = StorageHandler(container="rw")

handler_rw.write_csv_blob("output.csv", df)
handler_rw.write_parquet_blob("output.parquet", df)
handler_rw.write_excel_blob("output.xlsx", df)

# Stream LazyFrame directly to storage
handler_rw.sink_parquet_blob("output.parquet", lf)

# Upload any file (JSON, YAML, images, etc.)
handler_rw.upload_blob("config.json", "./local/config.json")

# List blobs
blobs = handler.list_blobs()    # Returns List[BlobInfo]
handler.download_blob("file")   # Download to local ./data/
handler_rw.delete_blob("file")  # Delete a blob or folder
handler_rw.move_blob("old.csv", "archive/old.csv")  # Move/rename

Configuration

Environment Variables

Set STORAGE_ENV to control the storage backend:

Value	Behavior
local (default)	Uses local filesystem (./data/ directory)
staging	Uses Azure staging storage (reads from .env.staging)
prod	Uses Azure production storage (reads from .env)

Azure Configuration

For Azure storage, create a .env file (for prod) or .env.staging (for staging).

Important: Restrict permissions on your .env file so only you can access it:

chmod 600 .env

Example .env (production)

AZURE_STORAGE_ACCOUNT=<azure_storage_account>
AZURE_CONTAINER_NAME_READ_PROD=<azure_container_name_read_prod>
AZURE_CONTAINER_NAME_WRITE_PROD=<azure_container_name_write_prod>
AZURE_TENANT_ID=<azure_tenant_id>
AZURE_CLIENT_ID=<azure_client_id>

# SAS tokens (optional, if not using device code auth)
AZURE_STORAGE_SAS_TOKEN_READ=<azure_storage_sas_token_read>
AZURE_STORAGE_SAS_TOKEN_WRITE=<azure_storage_sas_token_write>

Example .env.staging

AZURE_STORAGE_ACCOUNT=<azure_storage_account>
AZURE_CONTAINER_NAME_READ_STAGING=<azure_container_name_read_staging>
AZURE_CONTAINER_NAME_WRITE_STAGING=<azure_container_name_write_staging>
AZURE_TENANT_ID=<azure_tenant_id>
AZURE_CLIENT_ID=<azure_client_id>

# SAS tokens (optional, if not using device code auth)
AZURE_STORAGE_SAS_TOKEN_READ=<azure_storage_sas_token_read>
AZURE_STORAGE_SAS_TOKEN_WRITE=<azure_storage_sas_token_write>

Authentication

Use io auth to manage authentication explicitly. Device-code login and status auto-discover the environment from your .env / .env.staging file, so --env is optional. SAS token and client credentials login require explicit --env:

# Device code login (default) — auto-discovers env from .env file
uv run io auth

# Device code login with explicit env
uv run io auth --env staging

# Check auth status across all methods
uv run io auth status

# SAS token login — requires --env (wrong env = wrong token)
uv run io auth login --method sas-token --env staging

# Client credentials — requires --env (for CI/CD pipelines)
uv run io auth login --method client-credentials --env prod

# Clear cached device code tokens
uv run io auth clear

# Force re-authentication (clears cache first)
uv run io auth login --force

The chosen auth method is saved to the session. Subsequent commands (ls, download, etc.) automatically use it.

Authentication Methods

1. Device Code Flow (recommended for development)

   • Interactive browser-based authentication
   • Tokens are cached locally for 90 days
   • No credentials stored in files

2. Client Credentials

   • For CI/CD pipelines
   • Use io auth login --method client-credentials to authenticate
   • Requires:
     • AZURE_TENANT_ID: Service principal's tenant ID
     • AZURE_CLIENT_ID_CONTAINER: Service principal's client ID
     • AZURE_CLIENT_SECRET_CONTAINER_PROD: Service principal's client secret for production
     • AZURE_CLIENT_SECRET_CONTAINER_STAGING: Service principal's client secret for staging

3. SAS Token

   • Use pre-generated tokens with limited validity
   • Good for sharing temporary access

CLI Tools

The package provides a unified io command with subcommands for all storage operations. All commands accept --env prod|staging to override STORAGE_ENV.

Command	Usage	Description
auth	uv run io auth [login|status|clear] [--method M] [--force]	Manage authentication
ls	uv run io ls [--container ro|rw|ro-rw]	List blobs in storage
download	uv run io download <blob_name> [--container ro|rw]	Download a blob to local data/ directory
upload	uv run io upload <blob_name> <file_path>	Upload a local file to the write container
rm	uv run io rm <path>	Delete a blob or folder from the write container
mv	uv run io mv <source> <dest> [--overwrite]	Move (rename) a blob within the write container
generate-sas	uv run io generate-sas [--container ro|rw|ro-rw] [--days N] [--update-env]	Generate Azure Storage SAS tokens

Examples

# Authenticate with device code flow (auto-discovers env)
uv run io auth

# Check authentication status (auto-discovers env)
uv run io auth status

# List blobs in local mode (./data/)
uv run io ls

# List both read and write containers (default --container ro-rw)
uv run io ls --env staging

# Download a blob to the local data/ directory
uv run io download path/to/file.csv --env staging

# Upload a local file to the write container
uv run io upload path/to/blob.csv ./local/file.csv --env staging

# Delete a blob or folder from the write container
uv run io rm path/to/blob.csv --env staging
uv run io rm old-folder/ --env staging

# Move a blob within the write container
uv run io mv data/old.csv archive/old.csv --env staging

# Overwrite an existing destination
uv run io mv data/old.csv data/existing.csv --env staging --overwrite

# Generate SAS tokens and update .env.staging file
uv run io generate-sas --env staging --update-env

# Generate only read container token, valid for 3 days
uv run io generate-sas --env prod --container ro --days 3

Session

When doing multiple operations on the same container, use session to save defaults so you don't have to repeat --env and --container on every command. Both --env and --container are required to start a session.

# Start a session — saves env and container
uv run io session --env staging --container rw

# Show active session
uv run io session

# Now these use the saved defaults
uv run io ls
uv run io rm old-folder/
uv run io mv data/old.csv archive/old.csv

# Explicit flags still override the session
uv run io ls --container ro

# End the session
uv run io session clear

The session is stored in .containerio/session.json in the current directory. It is project-local and git-ignored.

The session is also available programmatically via the Session class:

from containerio.session import Session

session = Session()
print(session.auth_method)  # e.g. 'client-credentials'
print(session.env)          # e.g. 'staging'
print(session.is_active)    # True if any session data exists

Options:

• --env: prod or staging (default: staging for sas; uses STORAGE_ENV for other commands)
• --container: ro, rw, or ro-rw (default: ro-rw)
• --days: Token validity in days, max 7 (default: 1)
• --update-env: Write tokens to .env file instead of printing

Note: SAS tokens are generated using a user delegation key which has a maximum validity of 7 days (Azure limitation). See Microsoft documentation for details.

API Reference

StorageHandler

Main class for storage operations.

StorageHandler(
    container: Literal["ro", "rw"] = "ro",
    auth_type: Optional[str] = None,
    session: Optional[Session] = None,
)

• auth_type=None: Falls back to session auth method, then device code flow
• auth_type="device-code": Device code flow (interactive)
• auth_type="sas-token": SAS token authentication
• auth_type="client-credentials": Client credentials (service principal)
• session=None: Creates a new Session (skipped in local mode). Pass an existing Session instance for dependency injection.
• container="ro": Uses the read container (AZURE_CONTAINER_NAME_READ_{ENV})
• container="rw": Uses the write container (AZURE_CONTAINER_NAME_WRITE_{ENV})

Read Methods

Method	Description
read_parquet_blob(blob_name)	Read parquet file, returns (DataFrame, metadata)
read_csv_blob(blob_name, **kwargs)	Read CSV file into DataFrame
read_excel_blob(blob_name, **kwargs)	Read Excel file into DataFrame
scan_parquet_blob(blob_name)	Lazy scan parquet, returns (LazyFrame, metadata)
scan_csv_blob(blob_name, **kwargs)	Lazy scan CSV file

Write Methods

Method	Description
write_parquet_blob(blob_name, df)	Write DataFrame to parquet
write_csv_blob(blob_name, df, **kwargs)	Write DataFrame to CSV
write_excel_blob(blob_name, df, **kwargs)	Write DataFrame to Excel
sink_parquet_blob(blob_name, lf)	Stream LazyFrame to parquet

Utility Methods

Method	Description
upload_blob(blob_name, file_path, progress_hook)	Upload a local file to storage
delete_blob(blob_name)	Delete a blob, file, or folder recursively. Returns List[str] of deleted paths
move_blob(source, destination, overwrite)	Move (rename) a blob within the same container
download_blob(blob_name, progress_hook)	Download blob to local data/ directory. Returns Optional[Path]
list_blobs()	List all blobs, returns List[BlobInfo]

AzureStorageConfig

Configuration dataclass for Azure storage settings.

from containerio import AzureStorageConfig

# Load from environment
config = AzureStorageConfig.from_environment()  # Uses STORAGE_ENV
config = AzureStorageConfig.from_environment(environment="staging")

# Access configuration
print(config.storage_account)
print(config.container_name_read)

Environment

Enum for storage environments.

from containerio import Environment

# Parse from string
env = Environment.from_string("prod")  # Environment.PROD

# Use enum directly
if env == Environment.LOCAL:
    print("Using local storage")

Development

# Clone and install
git clone ssh://[e-mail]/Alliance-SwissPass/py-containerio.git
cd py-containerio
uv sync --group dev

# Run tests
uv run pytest

# Lint and format
uv run ruff check
uv run ruff format

Version Management with git-sv

This project uses git-sv for semantic versioning and changelog generation.

The .gitsv/config.yaml file defines how versions are bumped based on commit messages.

1. Commit messages: Follow conventional commits format
2. Version bump: The CI pipeline automatically runs git-sv bump based on commit history
3. Changelog: The CI pipeline generates the changelog using git-sv changelog and pins it as a Git issue
4. Release: The CI pipeline automatically handles tagging and publishing

The CI pipeline will automatically:

• Bump the version based on commit history
• Generate and pin the changelog as a Git issue
• Build and publish the package to Forgejo
• Create a release with the changelog notes

Publishing

Via CI Pipeline (recommended)

The repository includes a Crow CI pipeline that automatically builds, tests, and publishes the package when a git tag is pushed:

2. Commit and push your changes
3. Create and push a git tag (e.g., git tag v1.0.0 && git push --tags)

The pipeline will:

• Run linting and tests
• Update the version in pyproject.toml automatically
• Generate changelog from git history using git-sv
• Build and publish the package to Forgejo and PyPI
• Create a release with changelog notes
• Push version updates back to the repository

Note: For CI publishing, ensure the UV_PUBLISH_TOKEN secret is configured in your Crow CI pipeline settings.

Security Best Practices

1. Never commit credentials: Always add .env* to your .gitignore file
2. Restrict file permissions: Use chmod 600 .env to limit access to your credentials
3. Use short-lived tokens: SAS tokens have a maximum validity of 7 days
4. Rotate secrets regularly: Especially for production environments and CI/CD pipelines
5. Use device code flow: For development to avoid storing credentials in files
6. Environment isolation: Use separate credentials for staging and production
7. Limit token scope: When generating SAS tokens, specify the minimum required permissions

References

• Delegate access with shared access signature
• Create user delegation SAS
• Get user delegation key

License

Alliance SwissPass https://allianceswisspass.ch