hip-cargo 0.1.4

Tools for generating Stimela cab definitions from Python functions

hip-cargo

hip-cargo is an attempt to liberate developers from maintaining their packages in cult-cargo. The core concept boils down to maintaining a lightweight package that only installs the stimela cabs required to run a linked and versioned containerized image of the package. This makes it possible to install the package alongside cult-cargo and include cabs into recipes using the syntax

_include:
  - (module.cabs)cab_name.yml

In principle, that's all there is to it. The hip-cargo package does not dictate how you should go about structuring your package. Instead, it serves as an example of how to design auto-documenting CLI interfaces using Typer with automated cab generation and containerisation. It provides utilities to convert function signatures into stimela cabs (and vice versa) for packages that mimic its structure.

Installation

pip install hip-cargo

Or for development:

git clone https://github.com/landmanbester/hip-cargo.git
cd hip-cargo
uv sync --group dev --group test
uv run pre-commit install

Key Principles

1. Separate CLI from implementation: Keep CLI modules lightweight with lazy imports. Keep them all in the src/mypackage/cli directory and define the CLI for each command in a separate file. Construct the main Typer app in src/mypackage/cli/__init__.py and register commands there.
2. Separate cabs directory at same level as cli: Use hip-cargo to auto-generate cabs into in src/mypackage/cabs/ directory with the generate_cabs.py script. There should be a separate src/mypackage/cli/mycommand.py file corresponding to each cab.
3. Single app, multiple commands: Use one Typer app that registers all commands. If you need a separate app you might as well create a separate repository for it.
4. Lazy imports: Import heavy dependencies (NumPy, JAX, Dask) only when executing
5. Linked GitHub package with container image: Maintain an up to date Dockerfile that installs the full package and use Docker (or Podman) to upload the image to the GitHub Container registry. Link this to your GitHub repository.

Quick Start

The following instructions provide a guide on how to structure a package for use with hip-cargo. Note that hip-cargo itself follows exactly this structure and will be used as the running example throughout. It provides three utility functions viz.

• generate-cabs: Generate cabs from Typer CLI definitions.
• generate-function: Generate a Typer CLI definition from a cab.
• init: Initialize and new project.

By default, hip-cargo installs a lightweight version of the package that only provides the CLI and the cab definitions required for using the linked container image with stimela. Upon installation, an executable called hip-cargo is added to the PATH. hip-cargo is a Typer command group containing multiple commands. Available commands can be listed using

hip-cargo --help

This should print something like the following

Documentation on each individual command can be obtained by calling help for the command e.g.

hip-cargo generate-cabs --help

The full package should be available as a container image on the GitHub Container Registry. The Dockerfile for the project should install the full package, not the lightweight version. This is used to build the container image that is uploaded to the registry. The image should be tagged with a version so that stimela knows how to match cab configuration to images. The following versioning schema is proposed:

• use semantic versioning for releases
• use latest tag for main/master branch
• use branch-name when developing new features

This can all be automated with pre-commit hooks and GitHub actions. Use pre-commit hooks to auto-generate cab definitions on each commit. See the publish-container workflow for an example of how to set up GitHub Actions for automation. We distinguish between two cases viz. initialising a project from scratch or converting an existing project.

Using hip-cargo to initialise a project

The hip-cargo init command scaffolds a complete project with CI/CD pipelines, containerisation, pre-commit hooks, and Stimela cab support. Run it with:

hip-cargo init --project-name my-project --github-user myuser

This creates a ready-to-use project directory with:

• src layout with separate cli/, core/, and cabs/ directories
• pyproject.toml (PEP 621 compliant) with uv as the build backend
• GitHub Actions workflows for CI, PyPI publishing, container publishing, and automated cab updates
• Pre-commit hooks for ruff formatting/linting and automatic cab regeneration
• Dockerfile for building container images uploaded to GitHub Container Registry
• tbump configuration with hooks for version bumping and cab regeneration
• License file (MIT, Apache-2.0, or BSD-3-Clause)
• An onboard command that prints step-by-step instructions for completing CI/CD setup

The generated project includes an onboard command that guides you through the remaining setup steps:

cd my-project
uv run my_project onboard

This prints instructions for:

1. Creating a GitHub repository (with gh CLI)
2. Setting up PyPI trusted publishing (OIDC, no API keys needed)
3. Creating a GitHub environment for publishing
4. Creating a GitHub App for automated cab update commits
5. Configuring branch protection with the App in the bypass list
6. Making your first release with tbump

Once setup is complete, you can delete the onboard command and start adding your own commands following the same pattern.

Init options

Option	Default	Description
--project-name	required	Hyphenated project name (e.g. my-project)
--github-user	required	GitHub username or organisation
--description	"A Python project"	Short project description
--author-name	from git config	Author name
--author-email	from git config	Author email
--cli-command	from project name	CLI entry point name
--initial-version	0.0.0	Starting version string
--license-type	MIT	License (MIT, Apache-2.0, BSD-3-Clause)
--cli-mode	multi	single (one command) or multi (subcommands)
--default-branch	main	Default git branch name
--project-dir	./<project-name>/	Output directory

Transitioning an existing package

To transition an existing package that already contains stimela cab definitions, it is probably easiest to manually create the required directory structure (see below) and to use the generate-function command to convert your cabs into CLI definitions. Do this for each cab separately and register the relevant commands in your CLI module's __init__.py. You might want to take a look at the template files and copy the necessary files across (or initialize a new blank project and just use those). This is currently a manual process, we might add an automation script (or skill) to do this in the future. Further details are provided below.

Package Structure

We recommend using uv as the package manager. Initialize your project with the following structure (again using hip-cargo as the example):

hip-cargo/
├── .github
│   ├── dependabot.yml
│   └── workflows
│       ├── ci.yml
│       ├── publish-container.yml
│       ├── publish.yml
│       └── update-cabs.yml
├── scripts                      # Automation scripts
│   └── generate_cabs.py
├── src
│   └── hip_cargo
│       ├── cabs                 # Generated cab definitions (YAML)
│       │   ├── __init__.py
│       │   ├── generate_cabs.yml
│       │   └── generate_function.yml
│       ├── cli                  # Lightweight CLI wrappers
│       │   ├── __init__.py
│       │   ├── generate_cabs.py
│       │   └── generate_function.py
│       ├── core                 # Core implementations (lazy-loaded)
│       │   ├── __init__.py
│       │   ├── generate_cabs.py
│       │   └── generate_function.py
│       ├── recipes              # Stimela recipes for running commands via stimela
│       │   ├── __init__.py
│       │   └── gen_cabs.yml
│       └── utils                # Shared utilities
│           ├── __init__.py
│           ├── cab_to_function.py
│           ├── decorators.py
│           ├── introspector.py
│           └── types.py         # ListInt, ListFloat, ListStr NewTypes + parsers
├── tests
│   ├── __init__.py
│   └── conftest.py
├── Dockerfile                   # For containerization
├── LICENSE                      # MIT or BSD3 license encouraged
├── .pre-commit-config.yaml      # You should use these if you don't already
├── .gitignore                   # make sure your .lock file is not ignored
├── pyproject.toml               # PEP 621 compliant
├── tbump.toml                   # this makes releases so much easier
└── README.md                    # project README

Python CLI

uv expects your modules to live in src/mypackage/. As an example, let's see what the generate-cabs command looks like

from pathlib import Path
from typing import Annotated, NewType

import typer

from hip_cargo import stimela_cab, stimela_output

Directory = NewType("Directory", Path)
File = NewType("File", Path)


@stimela_cab(
    name="generate_cabs",
    info="Generate Stimela cab definition from Python CLI function.",
)
@stimela_output(
    dtype="Directory",
    name="output-dir",
    info="Output directory for cab definition. The cab will have the exact same name as the command.",  # noqa: E501
)
def generate_cabs(
    module: Annotated[
        list[File],
        typer.Option(
            ...,
            parser=Path,
            help="CLI module path. "
            "Use wild card to generate cabs for multiple commands in module. "
            "For example, package/cli/*.",
        ),
    ],
    image: Annotated[
        str | None,
        typer.Option(
            help="Name of container image.",
        ),
    ] = None,
    output_dir: Annotated[
        Directory | None,
        typer.Option(
            parser=Path,
            help="Output directory for cab definition. The cab will have the exact same name as the command.",  # noqa: E501
        ),
    ] = None,
):
    """
    Generate Stimela cab definition from Python CLI function.
    """
    # Lazy import the core implementation
    from hip_cargo.core.generate_cabs import generate_cabs as generate_cabs_core  # noqa: E402

    # Call the core function with all parameters
    generate_cabs_core(
        module,
        image=image,
        output_dir=output_dir,
    )

Each CLI module should be a separate file and all modules need to be registered as commands inside src/cli/__init__.py. For hip-cargo, this is what it looks like

"""Lightweight CLI for hip-cargo."""

import typer

app = typer.Typer(
    name="hip-cargo",
    help="Tools for generating Stimela cab definitions from Python functions",
    no_args_is_help=True,
)


@app.callback()
def callback():
    """hip-cargo: a guide to designing self-documenting CLI interfaces using Typer + conversion utilities."""
    pass


# Register commands
from hip_cargo.cli.generate_cabs import generate_cabs  # noqa: E402
from hip_cargo.cli.generate_function import generate_function  # noqa: E402
from hip_cargo.cli.init import init  # noqa: E402

app.command(name="generate-cabs")(generate_cabs)
app.command(name="generate-function")(generate_function)
app.command(name="init")(init)

__all__ = ["app"]

So we have two commands registered. That's all we'll need for this demo.

Packaging

This is one of the core design principles. The package pyproject.toml needs to be PEP 621 compliant, and it needs to enable a lightweight mode by default but also specify what the full dependencies are. For hip-cargo, it looks like the following:

[project]
name = "hip-cargo"
version = "0.1.3"
description = "Tools for generating Stimela cab definitions from Python functions"
readme = "README.md"
requires-python = ">=3.10"
license = { text = "MIT" }
authors = [
    { name = "landmanbester", email = "[e-mail]" }
]
keywords = ["stimela", "typer", "cli", "yaml", "code-generation", "radio-astronomy"]
classifiers = [
    "Development Status :: 4 - Beta",
    "Intended Audience :: Developers",
    "Intended Audience :: Science/Research",
    "License :: OSI Approved :: MIT License",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12",
    "Topic :: Software Development :: Code Generators",
    "Topic :: Software Development :: Libraries :: Python Modules",
    "Topic :: Scientific/Engineering :: Astronomy",
]
dependencies = [
    "typer>=0.12.0",
    "pyyaml>=6.0",
    "typing-extensions>=4.15.0",
    "libcst==1.8.6",
]

[project.urls]
Homepage = "https://github.com/landmanbester/hip-cargo"
Repository = "https://github.com/landmanbester/hip-cargo"
"Bug Tracker" = "https://github.com/landmanbester/hip-cargo/issues"

[project.scripts]
hip-cargo = "hip_cargo.cli:app"

[build-system]
requires = ["uv_build>=0.8.3,<0.11.0"]
build-backend = "uv_build"

[tool.ruff]
line-length = 120
target-version = "py310"
extend-exclude = ["src/hip_cargo/templates"]

[tool.ruff.lint]
select = ["E", "F", "I", "N", "W"]
ignore = []

[tool.ruff.format]
quote-style = "double"
indent-style = "space"

[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py", "*_test.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]
addopts = [
    "--strict-markers",
    "--strict-config",
    "--verbose",
]
markers = [
    "unit: Unit tests",
    "integration: Integration tests",
    "slow: Tests that take more time to run",
]

[dependency-groups]
dev = [
    "pytest>=8.4.2",
    "ruff>=0.13.2",
    "tbump>=6.11.0",
    "pre-commit>=4.0.0",
    "ipdb"
]
test = [
    "pytest>=8.0.0",
    "pytest-cov>=5.0.0",
]

Container Images and GitHub Actions

For stimela to use your package in containerized environments, you should publish OCI container images to GitHub Container Registry (ghcr.io). This section shows how to automate this with GitHub Actions.

1. Create a Dockerfile

Add a Dockerfile at the root of your repository. For example:

FROM python:3.11-slim

WORKDIR /app

# Install uv for fast package installation
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv

# Copy package files
COPY pyproject.toml README.md ./
COPY src/ src/

# Install package with full dependencies using uv (much faster than pip)
RUN uv pip install --system --no-cache .

# Make CLI available
CMD ["hip-cargo", "--help"]

2. Automate Cab Creation and Containerisation

You can automate cab generation using pre-commit hooks. For example, you could define the following in your .pre-commit-config.yaml

repos:
  - repo: local
    hooks:
      - id: generate-cabs
        name: Generate Stimela cab definitions
        entry: python scripts/generate_cabs.py
        language: system
        always_run: true
        pass_filenames: false
        stages: [pre-commit]

This uses this script to generate cabs for all commands defined in your CLI module. You should be able reuse the GitHub action for hip-cargo in .github/workflows/update-cabs.yml to automate container creation for your project. The basic idea is to validate your cab definitions and then to build and push the container to the GHCR. The workflow will tag the container with the branch name if there is an open PR to your default branch. Once the PR is merged, an action is triggered to update the image name in the cab definitions and push a latest version to GHCR. Pushing semantically versioned tags will trigger the same workflow (this is where tbump is quite useful). In this case the image name is tagged with the version.

3. Link Container to GitHub Package

To associate the container image with your repository:

1. Automatic linking: If your workflow pushes to ghcr.io/username/repository-name, GitHub automatically creates a package linked to the repository.

2. Manual linking (if needed):

   • Go to your repository on GitHub
   • Navigate to the "Packages" section
   • Click on your container package
   • Click "Connect repository" in the sidebar
   • Select your repository from the dropdown

3. Set package visibility:

   • In the package settings, set visibility to "Public" for open-source projects
   • This allows stimela to pull images without authentication

4. Using the Container with stimela

Once published, users should be able to simply include the cab definitions in their recipes. This only requires installing the lightweight version of the package, so it shouldn't clash with any other packages, in particular stimela and cult-cargo. Use the following syntax to include a cab in a recipe

_include:
  - (mypackage.cabs)cab_name.yml

stimela will automatically pull the matching version based on the cab configuration. You could optionally provide stimela recipes inside your project (see src/hip_cargo/recipes, for example). If the lightweight version if the package is installed it should be possible to run these recipes directly using the syntax

stimela run 'mypackage.recipes::killer_recipe.yml' recipe_name option1=option1...

Type Inference

hip-cargo automatically recognizes custom stimela types. These should be created using typing.NewType. See the generate-cabs definition above for an example.

Decorators

@stimela_cab

Marks a function as a Stimela cab. (Probably incomplete but it's basically a dictionary mapping.)

• name: Cab name
• info: Description
• policies: Optional dict of cab-level policies

@stimela_output

Defines a stimela output supporting the following fields (Probably incomplete but it's basically a dictionary mapping):

• name: Output name (top level, one below cabs)
• dtype: Data type (File, Directory, MS, etc.)
• info: Help string
• required: Whether output is required (default: False)
• implicit: Just use what you would put in the cab definition for stimela
• policies: Parameter level policies provided as a dict. See stimela docs
• must_exist: Whether an output has to exist when the task finishes (default: False)
• mkdir: create the directory if it does not exist (default: False)
• path_policies: Path policies provided as a dict. See stimela docs

Note that the order is important if you want to implement a roundtrip test.

Features

• Automatic type inference from Python type hints
• Support for Typer Arguments (positional) and Options
• Multiple outputs automatically added to function signature if they are not implicit
• List types with automatic repeat: list policy
• First-class comma-separated list types (ListInt, ListFloat, ListStr) with built-in parsers
• Proper handling of default values and required parameters
• Full roundtrip preservation of inline comments (e.g., # noqa: E501)
• Optional {"stimela": {...}} metadata dict in Annotated type hints for Stimela-specific fields
• Project scaffolding with hip-cargo init including CI/CD, containerisation, and onboarding

Quirks

Comma-separated list types (ListInt, ListFloat, ListStr)

Typer (and Click underneath it) does not support variable-length lists as a single CLI option value. For example, --channels 1,2,3 cannot be directly typed as list[int] because Click sees the entire 1,2,3 as one string argument.

The standard Typer workaround is to repeat the flag (--channel 1 --channel 2 --channel 3), which maps to list[int] with Typer's repeat mechanism. However, this is inconvenient for parameters that naturally take comma-separated values and results in a CLI interface that is different from the stimela interface.

hip-cargo solves this with dedicated NewType wrappers defined in hip_cargo.utils.types:

from hip_cargo.utils.types import ListInt, parse_list_int

@stimela_cab(...)
def my_func(
    channels: Annotated[
        ListInt,
        typer.Option(parser=parse_list_int, help="Channel indices"),
    ],
):
    # channels is already list[int] at runtime — no manual splitting needed
    ...

• ListInt, ListFloat, and ListStr wrap str (so Typer sees a single string argument)
• Paired parser functions (parse_list_int, parse_list_float, parse_list_str) handle comma-splitting at the Click level, so the function body receives the already-parsed list
• The introspector maps these types to the correct Stimela dtypes (List[int], List[float], List[str])
• The reverse generator (generate-function) automatically uses these types when it encounters a List[int]/List[float]/List[str] dtype in a cab YAML

Custom Stimela types via NewType

Stimela has its own type system (File, Directory, MS, URI) that doesn't map 1:1 to Python types. We use typing.NewType to create thin wrappers around Path:

from typing import NewType
File = NewType("File", Path)

These NewTypes serve double duty: they're valid Python type hints for Typer, and hip-cargo introspects the name to produce the correct Stimela dtype in the cab YAML. For File and Directory types, you also need parser=Path in the typer.Option() so Click knows how to parse the string argument.

Ruff formatting and config_file

The generate-function command runs ruff check --fix and ruff format on generated code. Ruff infers first-party packages from the working directory, which affects import grouping (e.g., whether import typer and from hip_cargo... get a blank line between them). When a --config-file is provided, hip-cargo runs ruff from the config file's parent directory so that first-party detection matches the target project rather than wherever hip-cargo happens to be invoked from.

Development

This project uses:

• uv for dependency management
• ruff for linting and formatting
• typer for the CLI

Setting Up Development Environment

# Clone the repository
git clone https://github.com/landmanbester/hip-cargo.git
cd hip-cargo

# Install dependencies with development tools
uv sync --group dev --group test

# Install pre-commit hooks (recommended)
uv run pre-commit install

This will automatically run the hooks before each commit. If any checks fail, the commit will be blocked until you fix the issues.

Running Hooks Manually

You can run the hooks manually on all files:

# Run on all files
uv run pre-commit run --all-files

# Run on staged files only
uv run pre-commit run

Updating Hook Versions

To update hook versions to the latest:

uv run pre-commit autoupdate

Manual Code Quality Checks

If you prefer to run checks manually without pre-commit:

# Format code
uv run ruff format .

# Check and auto-fix linting issues
uv run ruff check . --fix

# Run tests
uv run pytest -v

Contributing Workflow

1. Create a feature branch:

   git checkout -b your-feature-name
   

2. Make your changes and ensure tests pass:

   uv run pytest -v
   

3. Format and lint (automatically done by pre-commit):

   git add .
   git commit -m "feat: your feature description"
   # Pre-commit hooks run automatically
   

4. Push and create a pull request:

   git push origin your-feature-name
   

License

MIT License