idi-ftm2j-shared

Shared AWS infrastructure for the FTM2J terminal ecosystem. Two independent Pulumi stacks — deploy bootstrap first, then shared.

pulumi-bootstrap — GitHub Actions OIDC

Provisions the account-level OIDC identity provider and the two IAM roles that all dsi-clinic repos use to authenticate with AWS from GitHub Actions.

Run locally. This stack must be deployed from a workstation with AWS credentials — it creates the very roles that CI uses, so CI cannot deploy it itself.

cd pulumi-bootstrap
pulumi login s3://idi-ftm2j-dev-pulumi-state/ftm2j-shared/bootstrap # Config passphrase in Bitwarden
pulumi stack select dev
pulumi preview
pulumi up

Roles created (per repository):

Each repository gets its own checks + deploy role pair, trust-scoped to repo:dsi-clinic/<repo> so a repo's workflows can assume only its own roles. The stack loops over the idi:repos list (defaulted in infra/config.py); to onboard a new repo, add its name to that list and re-deploy this stack, then read its ARNs from the checks_role_arns / deploy_role_arns stack outputs. See docs/onboarding-a-processor.md for the full processor checklist.

pulumi — Shared Infrastructure

Provisions the AWS resources shared across all FTM2J processor pipelines. Individual processor stacks reference these outputs rather than creating their own copies.

cd pulumi
pulumi stack select dev
pulumi preview
pulumi up

Resources:

Stack outputs consumed by downstream processor stacks:

processor_bucket_name
processor_bucket_arn
s3_endpoint_id
s3_endpoint_arn
dlq_url
dlq_arn

deploy.yml is path-filtered: version/publish jobs only run when src/** or pyproject.toml changed; the Pulumi deploy job only runs when pulumi-shared/** changed.

development + contributing

Install all dependency groups (includes dev tools: pytest, ruff):

uv sync --all-groups

tests

uv run pytest

linting + formatting

uv run ruff check .    # lint
uv run ruff format .   # format

code style

branching strategy + versioning

Onboarding a processor repo onto the shared CI/CD flow? See the end-to-end checklist in docs/onboarding-a-processor.md (workflow callers, branch protection, deploy key, value routing).

Two-branch model with short-lived issue branches.

long-lived branches

Both branches are protected. All changes occur via pull request.

short-lived branches

• issue-<number>-<slug> — feature, bug-fix, and chore work.
  • Branch from dev, PR back to dev.
  • While the PR is open, only checks.yml runs (lint, tests, security, Pulumi preview). Pushes to the issue branch do not bump the version or deploy.
  • On merge, the push to dev triggers deploy.yml: bumps the alpha version and deploys the dev stack.
  • Note: It is best to create branches with this naming convention as you will be able to manually deploy these branches for testing in the dev stack. See (#manual-deploys)
• Hotfix — urgent production fix.
  • Branch from main as issue-<number>-hotfix-<slug>, PR back to main.
  • After release, merge main back into dev (see Syncing main back into dev).

ci/cd pipelines

Validation and deployment are split across two workflows:

• checks.yml — runs on every PR, required before merge. Lint, tests, security scan, Pulumi preview.
• deploy.yml — runs on push to dev or main (i.e. after a merge). Computes/bumps version, tags, releases, deploys Pulumi, publishes to PyPI (main only, if the repo includes a package).

versioning

The committed version in pyproject.toml is always a stable release. Alpha versions are never committed — they are computed inside the dev deploy run and used only for that run's tag/release. Only main writes a version back to the repo.

The alpha base is uv version --bump patch --dry-run --short (the next stable target), with a<run-number> for ordering and +<short-sha> for traceability. The +<sha> local segment is fine because alphas are never published to PyPI. Because the committed version is already stable, main uses --bump patch (not --bump stable, which would be a no-op).

Loop prevention. The bump (main) and sync-back (dev) commits are committed as idi-deploy-bot, and deploy.yml's version/deploy-pulumi jobs skip whenever the run's head commit was committed by that bot. This breaks the deploy→commit→deploy cycle without [skip ci] — which is deliberately avoided because it suppresses all workflows for the commit (including the required PR checks, silently blocking dev→main PRs) and trips on the literal string appearing anywhere in a message.

Each successful deploy:

1. main only: commits the bumped pyproject.toml + uv.lock (as the deploy bot, so it doesn't re-trigger deploy.yml). dev commits nothing.
2. Pushes a vX.Y.Z[aN][+sha] git tag.
3. Creates a GitHub Release — pre-release on dev, stable on main.
4. On main: builds the wheel/sdist and publishes to PyPI (if the repo ships a package), then the sync-dev job merges main back into dev (see below).

development cycle

1. dev → issue → alpha release

                                    PR
issue-123-add-feature  ────────────────────────────────► dev
        ▲                                              │
        │ branch                                       │ push triggers deploy.yml
        │                                              ▼
       dev ◄──────────────────────────────────── 0.1.6a123+abc1234, ...
                        merge                    deployed to dev stack
                                                 (alpha not committed)

1. git switch dev && git pull
2. git switch -c issue-123-add-feature
3. Commit, push, open PR targeting dev. checks.yml runs.
4. Merge the PR (squash recommended). The push to dev triggers deploy.yml:
   • Computes an alpha version in-workflow (e.g. 0.1.6a123+abc1234) — nothing is committed back to dev.
   • Tags, creates a pre-release, deploys the dev Pulumi stack, publishes the image. PyPI publish is skipped.
5. Each merge into dev produces a fresh alpha keyed to its run number and commit SHA. They all target the same next-patch base (e.g. 0.1.6a124+def5678, 0.1.6a125+...) until a stable release on main advances the base.

2. dev → main → stable release

dev (0.1.6a*) ───────── PR ─────────► main
 ▲                                      │ push triggers deploy.yml
 │                                      ▼
  ◄────────────────────────────────── 0.1.6 (stable)
              sync-dev job            deployed to prod stack
              (automatic)             published to PyPI

1. When dev is ready to ship, open a PR from dev → main. checks.yml runs against the prod Pulumi stack preview. (Because alphas are no longer committed to dev, the PR head is a normal commit and the required checks run.)
2. Review and merge. Do not squash — preserve history so release notes capture every change. A merge commit is fine.
3. The push to main triggers deploy.yml:
   • uv version --bump patch advances the committed stable version (0.1.5 → 0.1.6) and commits it to main as the deploy bot.
   • Tags v0.1.6, creates a stable GitHub Release, deploys the prod Pulumi stack, publishes to PyPI (if applicable).

3. syncing main back into dev

This is automatic: after a stable release, the sync-dev job in deploy.yml merges main back into dev (a direct push, committed by the deploy bot) so dev's pyproject.toml reflects the released stable version. Because the push is from idi-deploy-bot, deploy.yml's guards skip it and it doesn't re-trigger Deploy on dev.

The sync-dev push requires DEPLOY_KEY to be allow-listed in dev's branch protection.

If you ever need to sync manually (e.g. a hotfix landed directly on main):

git switch main && git pull
git switch dev && git pull
git merge main          # bring in the stable bump commit + any hotfixes
git push

The next push to dev then produces an alpha targeting the following patch (e.g. 0.1.7a*) above the just-released 0.1.6. On a pyproject.toml conflict, keep main's stable version.

manual deploys

deploy.yml accepts workflow_dispatch:

• From dev it deploys the dev stack.
• From main it deploys the prod stack.

Use this to redeploy Pulumi without a code change (e.g. after rotating a secret). Version/publish jobs stay gated on src/** changes.

summary

• dev is the only place new work lands; every merge produces an alpha.
• main cuts stable releases from whatever alpha dev is on.
• After every release on main, merge main back into dev.

branch protection rules

Branch protection, merge-method, deploy-key, and required-check conventions are documented once in docs/onboarding-a-processor.md (§2 Repo settings / branch protection, §3 Deploy key setup) — the same rulesets apply to this repo and every processor.