podpack

A framework for building web sites out of pluggable apps, together with the container substrate that runs them.

A site is a config file plus a list of installed apps. podpack supplies the application factory, the app registry, the template search order and the migration wiring. The site supplies its own chrome and its app list. Apps ship as ordinary Python packages and are installed by name:

[site]
name = "example.com"
apps = ["podpack_notes"]     # an app installed from its own repository

podpack itself installs no app — a repository that installed one would be a site. For a running example see ~/sites/podpack-demo, and creating-a-site.md for how to build one. The other side of the contract — writing, running, testing and shipping an app — is writing-an-app.md.

Adding an already-installed feature to a running site requires adding a line in that file and a restart — no code change, no rebuild, and no change to compose.yaml. A rebuild is only required to install new apps.

One site per instance. podpack builds a single site. It does not serve several domains from one process and there is no host-based routing; running two sites means two deployments — same packages, different config and different containers. That limit is deliberate, and it is what buys the simplicity elsewhere: one db.metadata, one alembic history, and one app list to reason about, rather than a registry keyed by hostname and a migration story per tenant.

This README says how to use podpack. For why it is the way it is — what forced each choice, what it cost, and what was rejected — see the architectural decision records.

The container suite is arranged so that no state and no host-specific setting lives inside a container: persistent state is bind-mounted from $HOST_DATA_DIR, host-specific configuration read-only from ./config, and secrets arrive through the environment. Promotion to a real host is an edit of .env alone.

Quick start

From the root of this repository:

./scripts/prepare-host-dirs.sh && ./scripts/up.sh

The first creates .env and secrets.env from their examples, with working lab values, and makes the host directories. The second always rebuilds — see Changing things for why that is the safe default.

Then visit http://localhost:8458/, or check the endpoint the container healthcheck uses:

curl -s localhost:8458/healthz

The richer view is /_status, and it answers operators only:

curl -s localhost:8458/_status     # 404 until you are one

An operator is a member of the admin role. Login is podpack's since ADR-0033, so every site can have one — including this lab, which before that record could never show this route at all. Three commands make you one; see The first administrator.

The 404 is deliberate — whether a site is a podpack site at all is not worth publishing — but it makes a refusal and a missing route look identical, so podpack says why at boot until somebody holds the role:

WARNING podpack: no 'admin' role exists, so /_status will answer 404 to
everyone -- including you. Create it and grant it: ...

That route reports the config file it read, the commit the image was built from, every installed app with the import name it came from and its data and log directories, and which database, role and schema it is actually connected as. If a mount or a grant is wrong, it says so.

The import name is worth having in front of you, because it is routinely not the app's own name — podpack_notes is what apps lists, and notes is what keys [site.mounts], [apps.<name>] and the directories on disk.

It also reports anything no installed app answers for:

"unclaimed": {
  "data": ["retired_app"],
  "logs": ["retired_app"],
  "tables": ["retired_app_notes"]
}

Normally all three are empty. The roots hold one subdirectory per installed app and nothing else, and every table belongs to some app. They drift legitimately, though: removing an app from apps deliberately does not delete its data or drop its tables, since uninstalling a feature should not destroy what it was holding. Reported rather than removed — deleting data because a config line changed would be the wrong instinct — so the answer to "do I have redundant components installed?" is visible.

tables is read from the database rather than from db.metadata, for the same reason the roots are read from disk: what a site declares and what it has are different things, and the gap is the entire point. Alembic's own alembic_version is excluded, belonging to the migration history rather than to any app. A table appearing here that you did not expect usually means either a retired app, or an app that never declared what it needs — see needs_tables.

Shut down with podman compose down, and come back with podman compose up -d — not start; see Stopping and starting. Host storage survives either way; see Starting over.

---

The plugin API

An app is a package exposing one module-level site_app. Everything else is convention. This section is the reference; writing-an-app.md is the worked guide to building one.

# myapp/__init__.py
from podpack import Section, SiteApp

from .views import blueprint      # Blueprint("myapp", __name__, ...)

site_app = SiteApp(
    blueprint=blueprint,
    url_prefix="/myapp",
    nav=(Section("My App", "myapp.index"),),
)

Field	Meaning
blueprint	An ordinary Flask Blueprint. Give it template_folder="templates" if it has templates.
url_prefix	Where the app asks to be mounted. None means the site root, and the site can overrule it — see below.
nav	Section(label, endpoint) entries contributed to the site's navigation, in installation order.
init	Optional callable(app), run before the blueprint is registered, for config keys and services.
needs_tables	Tables this app reads or writes but does not define. Several apps may need one table. Checked at boot: a table nothing installed defines is a missing dependency and the site refuses to start.
defines_tables	Tables it defines that SQLAlchemy's mapper registry cannot see — an association table built with a bare db.Table, say. Almost always empty; what an app defines is otherwise read from the registry, which is a fact rather than a declaration.
needs_secrets	Environment variables it cannot run without, checked at boot with podpack's own and the site's.

Two methods are optional overrides rather than fields, because an app that wants them usually wants both and usually has state to consult:

class Notes(SiteApp):
    def healthz(self):
        return Health(ok=store.reachable(), detail="mongodb unreachable")

    def status(self):
        return {"queued": store.pending()}

site_app = Notes(blueprint=blueprint, url_prefix="/notes")

Method	What podpack does with it
healthz()	Called on every /healthz. Returns Health(ok, detail, fatal), or None for "not reported" — which is not the same as healthy.
status()	Called on every /_status. Returns a mapping, reported under that app's reported key.

Both are called in an app context, both may do I/O, and podpack reports how long healthz() took so a slow check is visible rather than mysterious. An exception in either is caught and reported, never propagated: a health check is the last thing that should be able to take a site down, and a diagnostic that needs diagnosing is worth nothing.

A failing app does not by itself make the site unhealthy. /healthz stays 200 and names the app, because the container healthcheck gates the whole stack on that answer — one broken feature must not stop a site serving the rest. An app whose absence genuinely leaves the site pointless says fatal=True and gets a 503.

Install it by adding its import name to apps in the site's config file. Apps are installed in the order listed: nav entries appear in that order, and an app's init may rely on a service an earlier one registered.

Installing an app, and enabling one

These are two different operations, and only the second is free.

Enabling an app already present in the image is a line in app.toml and a restart — no code change, no rebuild, no compose change. That is the claim this framework is built around.

Installing one that is not yet in the image means putting the distribution there, which is a dependency change and a rebuild:

uv add "pp-pdf @ git+https://github.com/…/pp-pdf"   # records it in uv.lock
podman compose up -d --build                        # bakes it into the image

...and then the line in app.toml. podpack itself is indifferent to how the distribution arrived — apps = ["pp_pdf"] is an import name, and the registry only does import_module. An index, a git repository, a direct URL and a local path are all the same to it.

Only one of those sources asks for a tool the image would not otherwise have: git, because uv shells out to it. That is why the build stage installs one — see The image. A local path needs the source inside the build context, which a bind mount does not provide.

Index and URL installs need no extra tool, but that is not the same as needing nothing: the builder is python:3.12-slim and has no C compiler, so a dependency that resolves to an sdist needing compilation fails there whatever its source. Wheels are fine; anything that has to be built is not, until a toolchain is added.

What a site wires for itself

Some things belong to the site rather than to any one feature — mail, login, session policy. They are not apps, so they do not go in the apps list: pass a callable(app) as init instead.

# holdenweb/__init__.py -- what gunicorn is pointed at
import podpack

def create_app():
    return podpack.create_app(site_package="holdenweb", init=_wire)

def _wire(app):
    # `app_config` needs an app context and podpack pushes none before calling
    # init; and with no request to resolve an app from, the name is required.
    with app.app_context():
        settings = podpack.app_config("mail")      # from [apps.mail] or your own table
    app.config.update(MAIL_SERVER=settings["server"])
    mail.init_app(app)
    security.init_app(app, user_datastore)

It runs after the site's config is loaded and before the apps are installed, so an app's own init can rely on a service the site registered. Without the app_context() the site does not boot — it fails with RuntimeError: Working outside of application context., and an app's own SiteApp.init is subject to exactly the same rule.

The reason these are not apps is worth knowing, because writing shims to make them look like apps is a natural first thought: flask-mailman and flask-paranoid register no blueprint at all, and flask-security brings its own, created inside init_app. A SiteApp is built around a blueprint, so a shim would mean inventing one — and then inheriting a template namespace, a data directory and a log directory that nothing uses. See ADR-0025.

The app's name is its blueprint's name

site_app.name is derived, not declared, and it identifies the app everywhere it needs identifying: its template namespace, its data and log directories, and its section of the site's config file. So name the blueprint carefully — that is the decision.

It reads from the blueprint because that name is already the app's public identity: it prefixes every endpoint, and so appears in every url_for and every nav entry. It is also what podpack resolves an app from during a request, through request.blueprint.

A separate name field would be a copy of that, and a copy can drift. When there were two, nothing detected them disagreeing — the registry created and chowned one directory while the views read and wrote another, and app_config() quietly returned an empty dict, with nothing raised at boot or in the request.

Name the distribution podpack-<app name>

A convention, not a mechanism. An app's distribution should be called podpack- plus the app's own name — podpack-notes, podpack-pdf — in the way pytest-* and flask-* packages are. It makes an app findable on an index and tells a reader at a glance what a package is for.

podpack does not discover apps by scanning for that prefix, and the reasons are worth recording so the idea does not get reinvented:

• It would not remove anything. Scanning finds what is installed; it does not install it. The distribution is only present because the lockfile put it there, so its dependency entry is needed either way. What a scan would replace is the apps list — the other list.
• And the apps list is the part doing the work. It decides what is enabled, without a rebuild, and in what order. Discovery-by-presence means in-the-image equals switched-on, so turning a feature off becomes a rebuild, and ordering — which nav and init both depend on — is gone.
• It would fail quietly where it is needed most. An editable install, which is how you work on an app locally, need not register the module names such a scan reads. Discovery would work in the built image and find nothing on the bench.

If the app list ever does become a chore, the answer is entry points, not a name prefix: they impose no naming, work for a distribution called anything, and pp-pdf already ships one. That is the hybrid worth building — entry points for discovery, the config list for ordering and enablement.

Where an app lands is the site's decision

The app list decides whether a feature is installed. The shape of the address space stays the site's, so url_prefix is a request rather than a claim, and a site overrules it in a table of its own:

[site.mounts]
myapp = "/tools/myapp"

It lives under [site] rather than in [apps.myapp] because it is site policy and not app configuration: the app takes no part in the decision, and so never sees it — app_config() returns only what the app itself is meant to read. Two consequences worth knowing:

• The key is the app's name, which is its blueprint's name, and that is not always the import name in apps. podpack_notes is imported; it answers to notes.
• Naming an app that is not installed is a boot failure. Keeping mounts in their own table means the two can drift, and a stray entry would otherwise be silent — leaving the app at the address it asked for, which is exactly the address the site said it did not want.

Only apps being moved need an entry, so the table doubles as the site's map of everywhere it has chosen to put something.

Nothing else needs saying — not by the app, and not by the site. A Section names an endpoint, not a path, so the navigation resolves through url_for as the chrome renders and follows the app wherever it ends up. That is also why an entry naming an endpoint no view provides is a boot failure: a bad one would break url_for in the chrome and take out every page on the site, not just the page it points at, so it is worth refusing to start over.

Templates

Put templates under templates/<name>/, and refer to them the same way:

render_template("myapp/index.html")

The namespace is what stops two installed apps colliding on index.html. The search order is:

site templates  ->  app templates  ->  podpack defaults

Flask already searches the application's template folder before any blueprint's, and the site is the application — so a site overrides any app template simply by shipping one at the same namespaced path. podpack's own templates are appended last, which is why an app that extends base.html renders correctly on a site that has not written any chrome of its own yet.

Models

Put them in models.py. Nothing needs to import it:

# myapp/models.py
from podpack import db


class Thing(db.Model):
    __tablename__ = "things"
    id = db.Column(db.Integer, primary_key=True)

The registry imports that module while installing the app, and defining a db.Model subclass registers it on db.metadata as an import side effect. That import is the whole of model registration — and it is why migrations can see an app that the migration environment has never heard of. See Migrations for the consequence.

Name no schema. The application role's search_path points at the app schema it owns, so unqualified names land in the right place and alembic needs no schema configuration either.

Prefix __tablename__ with the app's name. Table names are the one identifier podpack does not namespace — templates, data and log directories and config sections all carry it, db.metadata is one flat namespace shared by every installed app. podpack warns as it installs an app whose table names its own name does not prefix, and refuses to boot a site where two apps claim the same one, naming both. /_status reports which app owns which table.

Data and logs

Every installed app gets a subdirectory of the host-mounted roots, named after the app:

<data root>/<name>/     persistent data the app owns
<log root>/<name>/      logs it writes

Uninstalling an app leaves its directories alone, so nothing is lost by taking a feature out of the app list and putting it back. /_status lists what is left behind under unclaimed, so retired data stays visible instead of merely present.

Resolve them with podpack.paths.data_dir() and log_dir(), which default to the app handling the current request. An app never builds these paths itself, so moving them at deployment time is a change to the environment and nothing else — and installing an app never requires a change to compose.yaml, because the roots are mounted and podpack creates the per-app directories inside them.

File logging comes free: podpack attaches a handler to the app's package logger, so logging.getLogger(__name__) inside the app writes to <name>.log as well as to stdout.

Shipping data with an app

An app may ship a data/ directory inside its package. On install, podpack copies it into that app's host data directory only if the target is empty.

That gives the same semantics as the database bootstrap in db-init/: "the first time on this machine", not "every time the container is recreated". Re-arming it means deleting the app's host data directory. The app then reads the host copy at runtime, so editing a shipped file on the host changes behaviour with no rebuild — the same property the mounted config files have.

Configuration

Each app gets a namespace of its own in the site's config file:

[apps.myapp]
page_size = 20

Read it with podpack.app_config(), which defaults to the app serving the current request. podpack never has to know what any of these settings mean.

Secrets do not go here. The split throughout is: non-secret settings that vary per host go in config/; secrets go in the environment. Config files are version-controllable and reviewable; .env is not committed.

---

Migrations

One alembic history for the whole site. The metadata alembic compares against is built by importing the models of every app the site configuration says is installed, so migrations follow the app list.

Generating a revision happens on the host, because the result is a file that belongs in the repository. Autogenerate compares the models against a live database, so this is the one job that wants the database port — which the suite does not publish by default (see ADR-0027). Ask for it, for as long as it takes:

podman compose --profile postgres-port up -d postgres-port

export PODPACK_CONFIG=config/app.toml
export SQLALCHEMY_DATABASE_URI=postgresql+psycopg2://holdenweb_app:…@127.0.0.1:5433/holdenweb
uv run alembic revision --autogenerate -m "what changed"

podman compose --profile postgres-port rm -sf postgres-port

POSTGRES_HOST_PORT=5439 podman compose --profile postgres-port up -d postgres-port picks a different number for one use, since a shell variable beats the one in .env.

Applying one happens in the container, automatically at startup, or by hand:

podman compose run --rm migrate alembic upgrade head
podman compose run --rm migrate alembic current

Do not reach for podman compose run --rm migrate alembic revision --autogenerate. It fails, and twice over: /app/alembic/versions is root-owned while the image runs as uid 10001, so alembic does the whole comparison and then dies on the final write with PermissionError; and even given permission, the file would be destroyed with the --rm container instead of landing in the repository. The image's code being read-only to the process running it is the right arrangement — generating revisions is simply not a container's job.

Because a revision's directory does not say which app it came from, its message should. See alembic/versions/205fc0d0ce92_notes_app_initial_schema.py.

Building that metadata deliberately does not construct a Flask app. The factory needs a secret key and a database URI before it will run, and coupling migrations to it would make a broken factory a broken migration too.

It does hold every app to the plugin contract even so, and the reason is the ordering rather than anything about migrations: migrate gates web with service_completed_successfully. While a module with no site_app was accepted here, that gate passed and the site's real failure surfaced in web — one service after the cause, so the logs blamed whatever came next. Checking costs no Flask app.

The footgun: autogenerate sees only the apps that are enabled

Run --autogenerate with an app missing from apps and alembic will faithfully propose dropping that app's tables, because from where it is standing they are tables no app claims. This is checked behaviour, not a theoretical risk.

Always autogenerate against the full app list. Django avoids this with per-app migration directories; podpack has one history, and per-app histories (version_locations plus branch_labels) are the answer if this ever becomes painful enough to be worth the extra heads to reason about.

Adopting an existing database

If the tables already exist, generate the revision against a scratch database and then baseline the real one rather than trying to apply it:

alembic stamp head
alembic check     # should report no new upgrade operations

---

The container substrate

Two ways to run a site

The same site, the same config/app.toml, the same app list and the same alembic history — run two ways:

./scripts/up.sh     # containers: podman compose, gunicorn, the real shape
./scripts/dev.sh    # local: no containers at all, Flask's dev server

dev.sh reads dev.env (made from dev.env.example on first run, and gitignored), applies the migration history, and starts the development server. It expects a natively installed PostgreSQL — not SQLite, and that is the one place this costs you a setup step:

createuser --pwprompt mysite_app
createdb --owner mysite_app mysite

podpack will not run those for you. The server is yours, and a run script that issued DDL against it would be doing something you had not asked for — so dev.sh checks, and prints exactly those two lines when it cannot connect.

Why not SQLite, when it needs no setup at all? Because alembic revision --autogenerate run against SQLite produces revisions PostgreSQL may reject. This project has the scar: a baseline written that way used ALTER COLUMN, which is PostgreSQL-only syntax and fatal on SQLite, and the mismatch surfaced only when a fresh container database was built months later. Authoring against the engine you deploy on is the whole point of authoring on the host (ADR-0011). SQLite remains right for tests, which create their schema directly and author nothing.

What the local mode does not give you: gunicorn, a proxy, the image, and any backing service beyond the PostgreSQL you installed — so a site running mongodb has none locally unless it installs one, and dev.sh says so.

Core services

A site chooses its backing stores. compose.yaml is the base — the site, its storage and the migration gate — and every store is an overlay beside it, named in COMPOSE_FILE:

COMPOSE_FILE=compose.yaml:compose.postgres.yaml

Service	What it is
postgres	Required. The SQL database db, the alembic history and the site's login tables all live in it, and create_app refuses to start without a SQLALCHEMY_DATABASE_URI.
mongodb	Optional. For apps that store documents rather than rows.

What is optional about PostgreSQL is the container, not the database: a site may drop compose.postgres.yaml from COMPOSE_FILE by hand and point the URI at a managed instance, which is the Opalstack arrangement ADR-0015 anticipated.

uv run podpack substrate services                 # what this site runs
uv run podpack substrate services --add mongodb   # enable another

--add records the service, rewrites COMPOSE_FILE, and delivers its variables to .env and its secrets to secrets.env.example by the ordinary append rule. Three things stay yours, because podpack writes neither credentials nor host state: the entries in secrets.env, a run of ./scripts/prepare-host-dirs.sh, and podman compose down && ./scripts/up.sh.

Choosing services is the site owner's job, taken independently of the installed apps. An app cannot declare that it needs one — that would be a dependency-management problem where a sentence in a README will do. And there is no --remove: taking a store away is a decision about data rather than configuration, so edit COMPOSE_FILE yourself and nothing podpack owns will have destroyed anything.

Each service publishes no host port, and brings its own on-request forwarder under a profile of its own name (see Ports):

podman compose --profile mongodb-port up -d mongodb-port
MONGODB_HOST_PORT=27020 podman compose --profile mongodb-port up -d mongodb-port

SQL is the one store an app may assume. db, its single metadata and its one alembic history are core, so the alembic environment is in the base rather than in postgres's overlay — what is optional is the server, and a site may point SQLALCHEMY_DATABASE_URI at a managed PostgreSQL and run no container at all. A site running no SQL keeps every other part of podpack and has a migrate service with nothing to do.

Why overlays and not compose profiles, since profiles are what they look like they are for: a service outside an enabled profile is not absent but undefined, so web.depends_on: {postgres: …} invalidates the entire project the moment the profile is off. Overlays merge depends_on additively, which is how the ordering guarantees survive being optional. Measured, and recorded in ADR-0028.

Secrets, checked at boot

podpack refuses to start without SECRET_KEY, SQLALCHEMY_DATABASE_URI and SECURITY_PASSWORD_SALT, and reports every one it is missing in a single message rather than the first:

this site cannot start: not set: SECRET_KEY, SECURITY_PASSWORD_SALT, MAIL_PASSWORD.
Secrets come from the environment, which compose fills from secrets.env; a local
run gets them from dev.env via scripts/dev.sh.

All at once because one name per restart is one rebuild per name on a containerised deployment. Each name is reported with whoever wants it, since that is what tells you whether to supply the secret or to stop installing the thing that asked for it.

Two other parties can add to the list. A site, by name, in the host config:

[site]
secrets = ["MAIL_PASSWORD"]

and an app, in its own SiteApp — because the author knows what the app reads and the site owner installing it has no way to:

site_app = SiteApp(blueprint=bp, needs_secrets=frozenset({"MAPS_API_KEY"}))

Names in the file or the app, values in the environment — the same split as everywhere else. Declare only what the thing genuinely cannot run without: a key that turns on an optional feature belongs in [apps.<name>] config with a sensible absence, because naming it here makes the whole site refuse to start.

The apps' declarations are checked immediately after the apps are imported, which is the first moment they exist; podpack's own three are checked before anything reads them, which is earlier still. Both are long before the site serves.

An empty value counts as missing, because SECRET_KEY= in an env file is not a configured site. So does one still holding CHANGEME or an unsubstituted @@TOKEN@@: those mean a file was installed and never edited, which is not a value but the absence of one wearing a value's clothes — and they are worth catching precisely because a site boots on them quite happily and is wrong in a way nothing else reports.

The first administrator

/_status answers a member of the admin role and nobody else, and a fresh database has neither the role nor a user — so a new site's operator view is unreachable until three commands have been run. They are flask-security's own, not podpack's:

flask --app mysite users create you@example.com --active
flask --app mysite roles create admin
flask --app mysite roles add you@example.com admin

The password is prompted for, hidden and confirmed. --active matters: without it the account is created and cannot sign in, which looks exactly like a wrong password.

Login itself is podpack's, not yours to wire — ADR-0033. podpack.auth ships the User and Role models, the datastore and the is_admin predicate /_status asks, and create_app installs them. A site writes nothing: no models.py, no Security(), no predicate. Mail and session policy remain the site's, under ADR-0025.

The three commands above are still flask-security's rather than podpack's, because they exist, validate the identity through the registration form and resolve users the way the rest of flask-security does; a framework copy would only be a worse one.

podpack says so at boot when the role does not exist yet:

WARNING podpack: no 'admin' role exists, so /_status will answer 404 to
everyone -- including you. Create it and grant it: ...

That exists because the endpoint refuses with 404 rather than 403 on purpose, which makes a refusal and a missing route identical from outside. It cost an afternoon once.

A site with an unusual idea of who counts as an operator can still pass its own create_app(admin=…); the default is a default, not a fixture.

Where the substrate comes from

podpack substrate upgrade reads its files from the podpack installed in this site's environment, whatever its provenance — a PyPI wheel, a git rev, a sibling checkout. Distribution policy and the substrate mechanism are independent, so pinning a site to an unreleased commit and upgrading from it works exactly as upgrading from a release does.

To apply a version you have not installed, name the artefact:

podpack substrate status  --from dist/podpack-0.9.0-py3-none-any.whl
podpack substrate diff    --from dist/podpack-0.9.0-py3-none-any.whl
podpack substrate upgrade --from dist/podpack-0.9.0-py3-none-any.whl

A wheel, an sdist, or a checkout directory. The wheel is read in place — the engine asks a root only for root / name, .is_file() and .read_bytes(), and zipfile.Path provides all three — so nothing is extracted and nothing is installed.

Installing the wheel first is not equivalent, which is why this exists. uv run re-syncs the environment from the lockfile before running, so a uv pip install of a local wheel is undone before the command meant to use it: install 0.8.0 over a lockfile pinning 0.7.3, run uv run podpack --version, and you get 0.7.3 with only an Uninstalled 1 package line to say so.

substrate.json records the artefact's version, not the installed one. Writing one version's files while recording another's would leave the next status comparing against the wrong thing.

Publishing

export UV_PUBLISH_TOKEN=pypi-...        # the project-scoped token, day to day
python3 tools/publish.py --dry-run
python3 tools/publish.py

The token lives in the environment, never in an argument — arguments are visible in ps. uv publish reads UV_PUBLISH_TOKEN itself, so the script never handles the value.

Scope the token to this project. PyPI offers a project scope only for a project that already exists, so the first upload needs an account-scoped token — which is a good reason for the environment rather than a config file, because it can be supplied for that one command and never become the default:

UV_PUBLISH_TOKEN=pypi-account-scoped python3 tools/publish.py

Then create a project-scoped token, revoke the account-scoped one, and export the narrow one for everything after. (An inline assignment lands in shell history; prefix the line with a space if your shell is set to ignore those, or revoke the broad token straight afterwards — which you are doing anyway.)

On CI, trusted publishing avoids tokens entirely — pass --assume-credentials, since keyring and OIDC cannot be detected from here.

It builds into a clean dist/ and refuses to upload unless exactly one version is present there, that version matches pyproject.toml, the working tree is clean, the r<version> tag exists, and the version is not already on PyPI.

The single-version guard is the one worth knowing about: uv publish uploads dist/*, that directory accumulates, and this repository sat with podpack-0.4.0 in dist/ while pyproject.toml said 0.7.3. Publishing then would have shipped 0.4.0, and PyPI does not let you take a version back.

Getting it, and keeping it current

The substrate ships inside the podpack package, and a site installs it with one command rather than by copying files out of this repository:

uv run podpack substrate init      # lay it down, or adopt a hand-copied set
uv run podpack substrate status    # how every file relates to the installed podpack
uv run podpack substrate upgrade   # bring the copy forward after upgrading podpack
uv run podpack substrate diff      # what exactly differs, per file

init derives the site's package from pyproject.toml (override with --site-package and friends), renders the one parameterised line — the Containerfile's gunicorn factory — and records what it wrote in substrate.json, which the site commits. Run on a site that already copied the substrate by hand, it adopts in place: identical files baseline silently, edited ones are kept and reported.

upgrade is a three-way comparison per managed file, against the recorded baseline of what podpack rendered: files you have not touched take upstream fixes; files you edited are kept, and said so; a file changed on both sides gets podpack's version written beside it as <file>.new and an exit status of 1 — resolve each with --take-upstream PATH or --keep PATH. --take-upstream also discards an edit you made earlier, which is how a site adopted with local differences converges; either way your version is kept as <file>.orig, because nothing is ever clobbered. status --check exits 1 if an upgrade would act, which is the CI hook; a damaged substrate.json exits 2, so the two are never confused.

A file that resolves outside the site — because you pointed it, or the directory holding it, at a shared checkout — is reported as not managed here and left entirely alone, in every command. That is a settled fact rather than pending work, so it does not hold --check red for ever.

Configuration is different, by design: once delivered, .env.example, secrets.env.example and a live .env change only by the addition of new parameters — an upgrade appends variables this site has never been given (each offered exactly once, so deleting one is respected) and never rewrites a line. The live secrets.env is never written at all: a newly-required secret is reported for you to add by hand, because an appended lab default in that file would be a weak credential on its way to production.

Out of the command's reach, always: config/app.toml, alembic/versions/, pyproject.toml, the lockfile, your source, and anything in scripts/ it did not put there — though the seeded .gitignore suggests scratch/ for your own experiments and one-off utilities, so that scripts/ can hold only the two files podpack manages. See ADR-0026 for the full rules and what was rejected.

This repository's own root is a rendered instance of the packaged substrate — podpack is its own first consumer — and a test pins the two byte-identical.

Ports

Service	Host port	Notes
Flask	127.0.0.1:8458	8456 is the real site's local port; 8457 is the MongoDB lab
PostgreSQL	none	published only on request — see below

The web port is offset on purpose: a lab that silently binds the production port is a lab that will one day be mistaken for production. It binds to loopback only, so it is not reachable from the network; change it in .env if it clashes.

The database publishes nothing. Nothing inside the suite needs it — the app and the migration service reach postgres:5432 across the compose network — so the only beneficiary was the host, at the cost of a number every other deployment on the machine had to avoid. Two sites are otherwise isolated by SITE_NAME alone. Ask when you want it:

podman compose --profile postgres-port up -d postgres-port

POSTGRES_HOST_PORT=5439 podman compose --profile postgres-port up -d postgres-port

A shell variable beats .env, so the second form chooses a number for one use without editing a committed file. podman compose --profile postgres-port rm -sf postgres-port takes it away again. See ADR-0027.

Where everything lives

What	Host location	Container location
Database cluster	$HOST_DATA_DIR/postgres/pgdata	/var/lib/postgresql/data/pgdata
Per-app data	$HOST_DATA_DIR/apps/<name>	/var/lib/podpack/apps/<name>
Per-app logs	$HOST_LOG_DIR/apps/<name>	/var/log/podpack/apps/<name>
PostgreSQL log	$HOST_LOG_DIR/postgres/postgresql.log	/var/log/postgresql
Server settings	config/postgresql.conf	/etc/postgresql/postgresql.conf (ro)
Client authentication	config/pg_hba.conf	/etc/postgresql/pg_hba.conf (ro)
Username mapping	config/pg_ident.conf	/etc/postgresql/pg_ident.conf (ro)
Site settings	config/app.toml	/etc/podpack/app.toml (ro)
Per-host wiring	.env	environment variables
Credentials	secrets.env	environment variables

HOST_DATA_DIR and HOST_LOG_DIR default to ./hostdata and ./hostlogs (both gitignored) so the suite is self-contained. On a real host they become absolute — /srv/mysite/data, /var/log/mysite — and nothing else needs to change. (The container side of the table above says podpack, not the site's name, and stays that way: those paths belong to the framework and are the same in every deployment. The host side is yours.)

Apps live under an apps/ level rather than beside postgres/ so that the two ownership fixes cannot reach each other: a single recursive chown of the data root would take the database's data directory with it.

Why there are two environment files

They are split by what restoring them means, not by secrecy:

	.env	secrets.env
Contains	paths, ports, site name, worker count	credentials, SECRET_KEY, database identity
On a new host	edit it — that is what it is for	put it back verbatim
If it changes	nothing is lost	sessions void, or the site cannot reach its own data

Mixing them is what made restoring a manual step: the backup had to be hand-edited before it could be used, in exactly the procedure that should have none. A restore is now copy secrets.env, edit .env — and the file you must not touch is the one you never open.

Only .env is read for variable substitution, so compose.yaml never refers to a credential and stays safe to commit and to read. podman compose config is the exception worth knowing: it expands env_file contents into the environment it prints, so treat its output as being as sensitive as secrets.env itself.

The site names its own containers

SITE_NAME in .env gives the compose project and the image their names:

$ podman ps --format '{{.Names}}'
holdenweb-lab-postgres-1
holdenweb-lab-web-1

So two sites on one host cannot collide, and podman ps says which is which instead of showing two identically-named sets. A second site needs distinct ports as well — WEB_HOST_PORT and POSTGRES_HOST_PORT are per-deployment, and a clash fails at bind time with address already in use.

Keep SITE_NAME in step with name in config/app.toml. Compose cannot read TOML, which is the only reason the site's name is written twice.

Why the data directory is a sub-directory

PostgreSQL refuses to start unless its data directory is mode 0700, and the permissions of a bind mount point belong to the host — on macOS virtiofs they come out world-writable. So the host directory is mounted at /var/lib/postgresql/data and PGDATA points one level deeper, at .../data/pgdata, which initdb creates itself and therefore gets right:

$ ls -ld hostdata/postgres hostdata/postgres/pgdata
drwxr-xr-x  hostdata/postgres/          <- the mount point, host's business
drwx------  hostdata/postgres/pgdata/   <- created by initdb, 0700 as required

Do not create pgdata yourself; prepare-host-dirs.sh deliberately does not.

Changing things

When in doubt, rebuild. src/ is baked into the image, so editing framework code and then reaching for restart brings back the previous build and leaves the site behaving like the last commit — a confusing symptom with an unrelated cause. Rebuilding unconditionally costs about six seconds when nothing has changed, because layers are content-addressed and an untouched file invalidates nothing:

./scripts/up.sh          # always rebuilds, and stamps the commit into the image

That is the safe default. The narrower loops are worth knowing because they are faster and because they are what a real host does:

podman compose restart web       # after editing config/app.toml
podman compose restart postgres  # after editing config/postgresql.conf

# after editing config/pg_hba.conf only -- no restart needed.
# `-u postgres` is required: pg_ctl refuses to run as root.
podman compose exec -u postgres postgres pg_ctl reload

podman compose up -d             # after editing .env (recreates containers)

Editing a mounted config file needs no rebuild and no image change, which is exactly the behaviour you want on a real host. pg_hba.conf is the one that can be applied without even a restart. Anything under src/, alembic/ or the Containerfile needs a build.

Which commit is actually running

scripts/up.sh stamps the commit into the image. Read it from the container, which works on any site including this lab:

$ podman compose exec web printenv PODPACK_BUILD_COMMIT
a7cf297-dirty

/_status reports the same value under build_commit, on a site whose admin predicate answers for you — not on the lab, which has no login and so no operator.

Compare it with git rev-parse --short HEAD and the question "is the container running the code I am looking at?" has an exact answer rather than an inference from timestamps. A -dirty suffix means the image was built from an uncommitted tree, which is normal while working and worth noticing when it is not. Building by hand instead reports unknown.

Stopping and starting

These are two pairs, and mixing them is the easy mistake:

podman compose stop     # containers keep existing, merely stopped
podman compose start    # ...so they can be started again

podman compose down     # containers are REMOVED (network too)
podman compose up -d    # ...so coming back has to recreate them

start only starts containers that already exist. After a down there are none, and it fails with service "init-storage" has no container to start — which reads like a fault in the one-shot service but is only saying the container is gone. up -d is always safe: it creates whatever is missing and starts the rest.

Both routes leave host storage alone, so no data is lost either way.

On the way up, either command honours the depends_on gates — init-storage and migrate run again before web. That is safe by design: the chown is idempotent and alembic upgrade head has nothing to do when the schema is already current.

ALTER SYSTEM will fail, by design

Because postgresql.conf is mounted read-only from the host, ALTER SYSTEM cannot write to it. That is the intended trade: configuration belongs to the host and to version control, not to whoever last had a superuser session.

Note also that a config file outside the data directory means initdb's own generated postgresql.conf is ignored entirely — so anything you need must be set in config/postgresql.conf or left at PostgreSQL's built-in default. That is also why hba_file and ident_file are named explicitly there: they default to sitting beside the config file, and all three are mounted together.

Reading the logs

PostgreSQL is configured with logging_collector = on, writing to a file on the host, matching how it would be run in production — so podman logs shows little for it beyond startup:

tail -f hostlogs/postgres/postgresql.log

The site logs to stdout, and each app additionally to its own file:

podman compose logs -f web
tail -f hostlogs/apps/notes/notes.log

Talking to the database directly

The shortest route needs no port and no local psql:

podman compose exec postgres psql -U labadmin -d holdenweb

To use the host's own psql — or any other client — publish the port first (podman compose --profile postgres-port up -d postgres-port, see Ports):

PGPASSWORD=holdenweb-app-password psql -h 127.0.0.1 -p 5433 -U holdenweb_app -d holdenweb

holdenweb_app is the application role: it can log in, connect to one database, and owns one schema. The superuser credentials in secrets.env are used exactly once, by the bootstrap below, and are never given to the app.

First-run bootstrap

db-init/01-create-app-user.sh creates the least-privileged application role. It:

• creates the holdenweb_app login role and grants it CONNECT,
• creates a schema app owned by that role, so it can create its own tables without any privilege over the rest of the database,
• sets the role's search_path to that schema, so unqualified table names land there — which is why apps' models name no schema,
• revokes CREATE on public from PUBLIC, making the intent explicit.

The image runs that directory only while the data directory is empty — and since the data directory is on the host, that means "the first time you bring the suite up on this machine", not "every time the container is recreated".

Backing up, and proving it

Three scripts, and one command that tells them what to do.

./scripts/backup.sh          # take one
./scripts/verify-backup.sh   # prove the last one can be read
./scripts/restore.sh <dir>   # put it back

Nothing in them is site-specific. They ask podpack backup plan what this site is made of, so installing an app changes what a backup contains with no edit anywhere, and a site that enables MongoDB gets its documents dumped because the catalogue knows how, not because somebody remembered.

A backup holds the three things a site cannot be rebuilt without — its stores, its per-app data directories, and secrets.env — plus a manifest.txt recording the three facts that must agree on the way back in: the commit, the app list, and the alembic revision. The app list is there because podpack's schema is a function of it: one alembic history covers whichever apps are enabled, so a dump restored against a different list leaves tables no app claims, and the next autogenerate proposes dropping them.

The backup directory is secret-bearing. secrets.env goes in verbatim, so that a restore is copy one file, edit the other rather than a hand-edit under pressure. BACKUP_ROOT therefore defaults outside the working tree and the script refuses to write inside it.

BACKUP_ROOT   where backups go        default ~/backups/<SITE_NAME>

What an app has to declare

Usually nothing. podpack already knows where an app's files live and reads its tables from the mapper registry, so an app that stores things is archived correctly without saying a word.

The exception worth declaring is statelessness, because an empty directory is ambiguous — podpack-qrcode holds zero bytes because it streams every code it makes, and a mount that never arrived looks exactly the same:

site_app = SiteApp(blueprint=bp, backs_up=Backup(data=False))

That is a claim podpack checks. An app saying it stores nothing while its directory holds files is warned about at boot and reported on /_status — warned rather than refused, because the failure it describes is in the future and an outage would be now. See ADR-0035.

Rehearsing it

verify-backup.sh reads every archive right through without applying any of it, using the same tool that would restore it, in the same container. Run it straight after each backup: a truncated archive found tonight is a non-event, and the same archive found in six weeks is a disaster.

That is the cheap half. The other half is restore.sh into a scratch deployment, which is the only thing that proves the data comes back. A backup regime is a claim about the future, and the only evidence for it is a restore that has actually happened.

Starting over

podman compose down
rm -rf hostdata hostlogs
./scripts/prepare-host-dirs.sh
podman compose up -d

Deleting hostdata/postgres/pgdata is what re-arms the database bootstrap; deleting an app's directory under hostdata/apps/ re-arms its data seeding.

How the services fit together

init-storage → postgres (waits for healthy) → migrate → web.

init-storage is a throwaway root container that hands the bind-mounted host directories to the unprivileged uids the servers actually run as (999 for postgres, 10001 for the app). Without it the server cannot write to a host directory it does not own. It is not a privilege escalation: under rootless podman that "root" is your own user inside a namespace, and on macOS the ownership change is namespace-local — the host keeps its own ownership.

migrate runs alembic upgrade head once and exits, gated by service_completed_successfully, so web cannot start against a stale schema. Doing it here rather than in the application also removes a race: gunicorn starts several workers at once, and anything creating tables at boot means the losers crash on tables a sibling has just made.

The database healthcheck is pg_isready -U … -d … rather than a bare pg_isready. The flags matter: without them it reports the server is accepting connections before the bootstrap has finished creating the application's database, and everything downstream starts too early.

The web healthcheck runs container/healthcheck.py as a script file, not a python -c one-liner: podman splits ["CMD", ...] healthcheck arguments on whitespace, so an inline probe arrives mangled and dies with a SyntaxError — reporting the container unhealthy however well it is actually running.

The image

Containerfile builds in two stages, because three things are needed to build the virtual environment and none of them to run it:

Left behind in the builder	Why it is there	Weight
git	uv shells out to it for a dependency locked to a git source	104 MB
the uv binary	resolves and installs from the lockfile	47 MB
uv's download cache	populated as a side effect of uv sync	~44 MB

No dependency is locked to a git source yet, so git is currently groundwork rather than load-bearing: the build would succeed without it today. It is installed ahead of need because the first app installed straight from a repository would otherwise fail the build with "Git executable not found", which names nothing that would lead you here.

Together that is roughly half the image: 398 MB single-stage against 203 MB. The runtime stage copies the finished .venv, the source, the migration environment and the healthcheck, and nothing else.

Note that removing git in a later layer would not have worked. The layer that installed it still carries the files, and a deletion only adds another layer on top — the image gets slightly bigger, not smaller. Not shipping it is the only way to not ship it.

Both stages must use the same WORKDIR

A venv is tied to its absolute path twice over. Console-script shebangs carry the interpreter path, and the project is installed into it as an editable pointing at <workdir>/src — which is also why the runtime stage copies the source: the venv alone is not a complete installation.

So a venv built under one directory and copied to another is thoroughly broken, not subtly so. Built under /build and copied to /app:

$ gunicorn --version
sh: 1: gunicorn: not found          # exit 127 — reads like a PATH problem
$ python -c "import podpack"
ModuleNotFoundError: No module named 'podpack'

Neither message mentions the venv, which is what makes it worth knowing. It is the same trap as renaming the project directory on the host, where uv sync will not repair it either because it audits packages rather than scripts. There the fix is rm -rf .venv && uv sync --all-groups; here it is keeping the two WORKDIR lines identical.

In this file a mismatch mostly fails loudly instead: COPY --from=builder /app/.venv cannot find its source and the build stops. Only changing both paths to different values produces the broken image above.

Deploying to Opalstack

Opalstack's AlmaLinux 9 servers run rootless podman, so this suite deploys there essentially as it stands. The mapping:

Opalstack gives you	goes in
an Nginx Proxy Port app's port assignment	WEB_HOST_PORT in .env
the app directory ~/apps/<name>/	HOST_DATA_DIR, HOST_LOG_DIR in .env
an nginx in front of you, terminating TLS	PODPACK_PROXY_HOPS=1 in .env
the site domain	nowhere — see below

That is the whole of it, which is the point: the port a managed host allocates is exactly the kind of per-host fact .env exists for. Opalstack generates the nginx upstream to proxy your site to that port, so nothing above the container needs to know it.

Two things to watch, neither of them podpack's doing:

• podman-compose is what their tutorial uses, and it will not honour this suite's ordering. See Compose front-ends: the depends_on gates are load-bearing here and it ignores them. Use podman compose with the Compose v2 provider, or sequence the phases by hand.
• loginctl enable-linger <uid> is needed for containers to keep running when you are not logged in; their tutorial mentions it in passing.

The domain goes nowhere, and PODPACK_PROXY_HOPS is why

The instinct is to write the site's domain into its configuration. Don't: a managed host's domain binding belongs to its control panel, and a site told its own name goes wrong the moment somebody changes it there.

It does not need telling. Every URL a site builds is built inside the request that asked for it, and nginx passes the visitor's Host through — so the site answers correctly under whatever name reaches it, including one nobody knew about when it was deployed. What nginx does not pass through is the scheme: the visitor arrives over TLS and the proxy forwards plain HTTP, so without PODPACK_PROXY_HOPS=1 every absolute URL comes out http://.

Nothing looks broken when that is wrong. Every page serves; only absolute URLs are affected, and nginx redirects them to https anyway. The first sign is a password-reset mail carrying an http:// link — which is how it was found here — and the cost is that a recipient whose browser holds no HSTS entry sends the reset token in the path of a cleartext request. /_status reports what arrived and what the site concluded, so the setting can be confirmed without mailing anybody. ADR-0036 has the reasoning, including why this is an environment variable and not a key in config/app.toml.

base_url in config/app.toml is a different and narrower thing: the address to use where there is no request to ask — a cron job, a CLI command building a link with no browser on the other end. A site whose links are all built inside requests does not need it. Where a site does, it is the public URL, https://example.com, with no port: the allocated port is where the container listens, not how the world addresses the site.

Using the managed PostgreSQL instead

Opalstack provides a managed PostgreSQL 17, the same version this suite runs in a container. Swapping to it is deliberately small, because podpack learns about the database only from SQLALCHEMY_DATABASE_URI: drop the postgres service, its two init-storage mounts and the db-init/ mount, and repoint the URI. No application code and no migration changes.

Worth knowing what it costs, though. A container pins postgres:17 per deployment and upgrades when you decide; the managed instance is the server's, shared with everything else on it, and moves when the host moves. Keeping the container is the same instinct as mounting the config read-only — the version belongs to version control rather than to the machine.

Configuring a host

After cloning onto a host, one command replaces every manual edit:

python3 scripts/configure-host.py --port 8461

It writes .env and secrets.env at mode 0600, keeping every comment from the examples, and it settles by construction the things that used to be settled by hand:

secrets	generated per host, never the examples' lab values
the database URI	built from POSTGRES_APP_USER/PASSWORD/DB, so those four cannot disagree
SELinux	detected from /sys/fs/selinux/enforce, setting VOLUME_RW/VOLUME_RO
prerequisites	reports a non-v2 compose provider, a missing podman socket, or lingering being off

Generated values are [A-Za-z0-9_-] and nothing else, deliberately. Each one passes through compose's env reader, a shell, a PostgreSQL URI and — inside alembic — Python's configparser, and every one of those treats some punctuation as syntax. A password containing % once stopped a deployment dead, in configparser, before any connection was attempted.

It refuses to overwrite an existing .env or secrets.env. Those carry a running site's identity: the salt every stored password is keyed on, and the role the database was bootstrapped with. Regenerating them does not reconfigure a site, it loses it. --force exists for a site with no data yet.

Standard library only, and it imports nothing from podpack — it runs on the host's system Python before anything has been installed or built.

Then:

./scripts/prepare-host-dirs.sh && ./scripts/up.sh

Running on Linux

Two differences on a real Linux host:

• SELinux (RHEL, AlmaLinux, Fedora, CentOS Stream). Set both relabel variables in .env:

  VOLUME_RW=:Z
  VOLUME_RO=,z
  

  Every bind mount interpolates them, so nothing in compose.yaml is edited — which matters, because that file is substrate-managed and a hand edit would put the site in conflict on every upgrade for as long as it lives.

  Without them SELinux denies the container access, and the failure misleads: PostgreSQL starts and reports healthy, because initdb creates the cluster and the superuser regardless — but db-init is unreadable, so the application role and schema are never created, and it is the migrate service that fails. Measured on an AlmaLinux host.

• Ownership. init-storage handles it, but if you prefer to pre-create the directories yourself, prepare-host-dirs.sh does the equivalent podman unshare chown on Linux.

Compose front-ends

They are not interchangeable, and this suite needs podman compose.

• podman compose — delegates to Docker Compose v2, which honours depends_on conditions. Required, because every ordering guarantee here rests on them.
• podman-compose — starts the containers but silently ignores depends_on conditions. It also names containers with underscores (holdenweb-lab-pg_web_1) rather than hyphens, so never point the two front-ends at the same project without taking the stack down first.

The difference is not theoretical. The same file, one service sleeping five seconds and a second gated on its completion:

podman-compose:   ONCE-START 626   AFTER-START 626   ONCE-END 631   # gate ignored
podman compose:   ONCE-START 633   ONCE-END 638      AFTER-START 638  # gate honoured

Under podman-compose three guarantees quietly disappear: init-storage no longer precedes the servers, so the bind-mount ownership problem returns; web no longer waits for migrate, so the site can start against a schema that has not been created; and it no longer waits for PostgreSQL to be accepting connections. Nothing reports any of this — the stack simply comes up, and works or does not depending on timing.

An earlier version of this file said podman-compose "also works". It was inherited from the original lab and had never been tested.

Development

uv sync
uv run pytest
uv run mypy

The tests cover what the registry promises — that the app list is configuration rather than code, that models reach db.metadata, that template namespacing and site override both work, that data seeds once and re-arms on deletion, and that the migration environment needs no Flask app.

mypy is a dependency rather than something to remember, because annotations nobody checks are comments that look authoritative. It reads its settings from pyproject.toml and covers src/ and tests/ both. Two suppressions exist and both say why in place: db.Model, which flask-sqlalchemy builds at runtime, and one deliberate SiteApp(name=...) in a test that asserts the call is an error.

There is a MongoDB sibling of this substrate, near-identical in shape and on different ports so the two can run side by side. It stayed in the holdenweb.com working tree when this project was extracted.

Testing a change without releasing it

Releasing to test is a habit worth naming, because the cost is not the wait. A tag is permanent, a version number on PyPI can never be reused, and a release made to answer a question leaves a version in the history whose only reason for existing was that somebody wanted to try something. It also invites the mistake of treating the tag as the delivery mechanism when it is only the packaging one.

Almost nothing here needs publishing to be tested.

Library code. uv run pytest, and provoke-errors.py for the failures the guides quote. This is the whole of it for anything that does not touch a file a site copies.

Library behaviour in a running site. This repository is a substrate instance — test_the_repo_root_is_the_rendered_substrate exists to keep it one — so scripts/dev.sh runs podpack against a real PostgreSQL with no containers, and scripts/up.sh runs it in them. It installs no apps, which is the right default: what it exercises is the framework, the substrate and the container suite rather than anybody's blueprint.

Substrate files — the Containerfile, the compose files, scripts/, alembic/env.py, the examples. These reach a site through podpack substrate upgrade, which reads them from the installed podpack, and that is what makes a release look necessary. It is not. Every substrate subcommand takes --from, which reads a checkout, a wheel or an sdist instead:

# in a site, against podpack's working tree -- uncommitted edits included
uv run podpack substrate status --from ~/sites/podpack
uv run podpack substrate diff   --from ~/sites/podpack
uv run podpack substrate upgrade --from ~/sites/podpack

Measured: appending a line to substrate/data/scripts/backup.sh in a working tree, committing nothing and publishing nothing, makes a site report scripts/backup.sh update available immediately.

To rehearse against exactly what a release would ship rather than what the working tree holds, build the artefact and read it in place:

uv build
uv run podpack substrate status --from ~/sites/podpack/dist/podpack-*.whl

Both the wheel and the sdist work. What does not work is uv pip install of a local wheel: uv run re-syncs the environment from the lockfile first, so the install is silently undone before the command that was supposed to use it — measured, with only an Uninstalled 1 package line to say so. --from exists because of that.

A library change inside a container. The one case that genuinely needs something on a remote, because the build context holds only pyproject.toml and uv.lock when uv sync --frozen runs, so a local wheel cannot be reached without editing the Containerfile — which is substrate-managed and would then conflict on every upgrade. Point the site at a commit instead:

[tool.uv.sources]
podpack = { git = "https://github.com/holdenweb/podpack.git", rev = "<sha>" }

Then uv lock --upgrade-package podpack --refresh-package podpack. That needs a push and no release: no tag, no version consumed, nothing on an index that never forgets. The Containerfile's git layer exists for exactly this.

What is left that does need a release is the release path itself — that the tag fires the workflow, that trusted publishing still holds, that the artefact installs from PyPI. Worth exercising deliberately, and worth knowing that it is the only thing on that list.

A worked example of getting this wrong. The fix that made restore.sh work on Linux is a substrate file. It was tested, tagged, published as 0.9.0b3 and installed, and only then delivered to the site — where it arrives as scripts/restore.sh, committed to the site's repository and reaching the host over git. substrate upgrade --from would have produced the identical file with none of that. The release was not wasted, because the version floor it allows stops a later plain substrate upgrade putting the broken script back; but it was not on the path to testing, and treating it as though it were cost a tag on an orphaned commit and a publish that never fired.