traintraintrain

Runs your releases like a railway. Each version is a train that calls at a series of stops in order: a canary first, then progressively larger groups of platforms (regions, clusters, …). Each platform keeps its own timetable, so deploys only start during its service hours, in its own timezone. After each deploy the train dwells at the platform while it's inspected (health checks). On failure it rolls back and halts. A signal can hold the train at a stop until a person clears it.

traintraintrain does not deploy anything itself. You supply shell commands (helm, kubectl, terraform, CodeDeploy, etc.), and traintraintrain decides when and where to run them.

Vocabulary

Features

• Stops: a canary first, then progressively larger stops, serving up to parallel_platforms platforms at a time.
• Timetables per platform: e.g. "Mon–Thu 09:00–16:00 Europe/Dublin". DST is handled, service hours can cross midnight, and line closures can apply globally or to one timetable (holiday freezes, launch days).
• Dwell and inspection: after each deploy, the platform dwells while inspections run. On failure the train rolls back (the platform, the stop or the whole train) and halts.
• Signals: a stop can wait for someone to clear its signal before deploying.
• Resumable: the train survives crashes and restarts. Re-run the same command to continue, or retry just the failed platforms.
• Slack: one live-updating message per train, with details in its thread and a mention on failures.
• Web UI: a departure board where you can dispatch trains, clear signals, pull the brake and retry, with an audit trail of who did what.
• Small: one runtime dependency (PyYAML). The UI has no build step and needs no CDN.

Install

Requires Python 3.10+, on Linux or macOS.

pip install traintraintrain

This installs the traintraintrain command.

Or use the container image; see Docker and Kubernetes.

Quick start

traintraintrain -c examples/train.yaml timetable            # projected departure per platform
traintraintrain -c examples/train.yaml service              # upcoming service hours
traintraintrain -c examples/train.yaml depart --version v1.4.0 --previous-version v1.3.2
traintraintrain -c examples/train.yaml board --version v1.4.0
traintraintrain -c examples/train.yaml clear-signal --version v1.4.0 --stop atlantic
traintraintrain -c examples/train.yaml brake --version v1.4.0   # from another shell
traintraintrain -c examples/train.yaml ui      # web dashboard on http://127.0.0.1:8080

How a journey runs

1. The train calls at stops in order. It leaves a stop only after every platform there has finished dwelling. Then it waits out the next stop's headway, if one is set.
2. If a stop has signal: true, the train waits there until someone clears the signal.
3. At each stop, up to parallel_platforms platforms are served at once. Each platform:
   • is held at a signal until its timetable is in service. If that's further away than max_hold, the platform fails.
   • deploys by running deploy_command, which is subject to deploy_timeout.
   • dwells for dwell, running inspection_command every inspection_interval.
4. On any failure the train halts and serves no new platforms. Platforms already being served finish, and rollback runs according to rollback_scope:
   • none
   • platform: only the failed platform
   • stop: the failed stop
   • train: every platform updated on this journey
5. State is saved to .traintraintrain/<train>-<version>.json. Re-running the same command resumes from that state, and platforms that succeeded are skipped. Failed platforms need --retry-failed.

Config

See examples/train.yaml. timetable, dwell and deploy_timeout can be set per platform, per stop, or as train defaults (default_timetable, default_dwell, deploy_timeout). The most specific setting wins.

Global train.line_closures, such as a holiday freeze, apply to every timetable. A timetable without service_hours is always in service. The built-in timetable always is also available.

Commands are templates using {train} {platform} {stop} {version} {previous_version} plus any platform vars. The same values are exported as env vars (DT_PLATFORM, DT_STOP, DT_CLUSTER, …). Escape literal braces as {{ }}.

Signals

A stop with signal: true stops the train before deploying. The train's status becomes at_signal, and Slack posts a message that links to the UI if ui_url is set. The train proceeds once someone clears the signal, either in the web UI or with traintraintrain clear-signal --version V --stop S. The name of whoever cleared it is recorded. If you brake and then resume, a signal that was already cleared stays cleared.

Web UI

traintraintrain -c train.yaml ui --host 127.0.0.1 --port 8080

What you can see

• Departures, showing each train's status and progress.
• Route view. Each platform shows its status and any failure output. It also shows whether its timetable is in service, or when service resumes in the platform's own timezone.
• Events, with routine inspections hidden unless you ask for them.
• Live log of the journey.
• "Who did what", an audit trail of actions taken in the UI.
• Timetable, showing each platform's service right now and projected departures for a train leaving now.

What you can do

• New departure, with a dry-run option and a preview of the timetable.
• Clear a signal.
• Pull the emergency brake.
• Resume an interrupted or braked journey.
• Retry failed platforms.

How it works

• The UI only reads state files. Each train runs as a separate traintraintrain depart process, logging to .traintraintrain/<train>-<version>.log. The UI can restart without affecting trains en route.
• The brake and signals use the same marker files as the CLI.
• Only one process may drive a train at a time. A train whose process died shows as interrupted, and you can resume it.

Access and auth

• By default the UI listens only on localhost.
• Shared deployments: put the UI behind your SSO proxy (oauth2-proxy, IAP, etc.). The user name is taken from X-Forwarded-User / X-Auth-Request-User (or the matching ...-Email headers) and recorded on every action.
• Shared token: alternatively, set TRAINTRAINTRAIN_UI_TOKEN to require a token. Users then enter a name in the header, which is self-reported.
• Every POST must send X-Requested-With: traintraintrain (CSRF protection).
• Actions are appended to .traintraintrain/<train>.audit.jsonl.

Slack notifications

notifications:
  slack:
    channel: "#deploys"             # bot mode needs a channel plus $SLACK_BOT_TOKEN
    token_env: SLACK_BOT_TOKEN      # default
    webhook_url_env: SLACK_WEBHOOK_URL  # default; used when no bot token is set
    mention_on_failure: "<!here>"   # or "<!subteam^S0123ABCD>" / "<@U0123ABCD>"
    ui_url: https://traintraintrain.example.com  # signal messages link here
    events: [...]                   # optional filter, see below

Bot mode (recommended). Create a Slack app with the chat:write scope, install it, invite it to the channel and export SLACK_BOT_TOKEN.

• Each train gets one message in the channel. It is edited live with a per-stop status board, e.g. canary ✅ us-west-2 · atlantic 🚦 eu-west-1 🚀 us-east-1.
• Every event is posted as a reply in that message's thread.
• Failures, red signals and the final result are also broadcast to the channel.
• If you resume the train, updates go to the same thread.

Webhook mode. Export SLACK_WEBHOOK_URL (an incoming webhook). Each event is posted as its own message.

The available events are train_started, train_resumed, train_finished, stop_started, stop_completed, stop_at_signal, signal_cleared, platform_held, platform_deploying, platform_succeeded, platform_failed and platform_rolled_back. All are sent by default except platform_deploying. Failure messages include the last part of the failing command's output, plus the mention_on_failure mention.

If Slack is unavailable. A Slack error or a missing credential is logged as a warning and never stops a deploy.

• traintraintrain notify-test sends a test message so you can check the setup.
• depart --no-notify turns notifications off for one journey.
• Notifications are always off in --dry-run.

Docker and Kubernetes

Images are published to GitHub Container Registry for linux/amd64 and linux/arm64: ghcr.io/vishnut-procore/traintraintrain:<version>, plus :edge built from main.

# Web UI, with config mounted read-only and state on a volume
docker run -p 127.0.0.1:8080:8080 -e TRAINTRAINTRAIN_UI_TOKEN=change-me \
  -v ./config:/config:ro -v traintraintrain-state:/state ghcr.io/vishnut-procore/traintraintrain:0.1.1

# One train as a one-off job (exits 0 when it arrives, 1 if it fails or is braked)
docker run -v ./config:/config:ro -v traintraintrain-state:/state \
  ghcr.io/vishnut-procore/traintraintrain:0.1.1 depart --version v1.4.0 --previous-version v1.3.2

Add your deploy tools to the image. Your deploy_command, inspection_command and rollback_command run inside the container. Build your own image FROM this one with kubectl, helm, aws, … installed; examples/docker/Dockerfile.tools shows how.

How the image behaves

• Config and state: the config is read from /config/train.yaml (TRAINTRAINTRAIN_CONFIG), and progress is saved under /state (TRAINTRAINTRAIN_STATE_DIR, which overrides the config's state_dir). Keep /state on a persistent volume.
• Non-root: the image runs as user 10001. Bind-mounted state directories must be writable by that user.
• Health check: /healthz needs no token, and the image's HEALTHCHECK uses it.
• Stopping a container (docker stop, a pod shutting down) is graceful. A depart job pulls its train's emergency brake. The UI brakes the trains it dispatched and waits up to TRAINTRAINTRAIN_SHUTDOWN_GRACE seconds (default 20). The journey can then be resumed.
• Restarts are recoverable. The process driving a train writes a heartbeat to the state file. If a container is replaced mid-journey, the train shows as interrupted once its heartbeat is 90 seconds old, and you can resume it with no --force.
• Run one UI instance per state directory. Two instances could drive the same train twice.

Examples

• examples/docker/docker-compose.yml: the UI with a persistent volume. Run it with TRAINTRAINTRAIN_UI_TOKEN=change-me docker compose up --build.
• examples/kubernetes/ui.yaml: the UI as a single-replica Deployment (Recreate strategy) with a PersistentVolumeClaim, probes, a non-root security context and a Service.
• examples/kubernetes/job.yaml: one train per release as a Kubernetes Job, for CD pipelines.

Running it in CI

A train can take days to complete its journey while it waits for service hours, so run it as a long-lived job, such as a Kubernetes Job (see above) or a self-hosted runner, with state_dir on persistent storage. Alternatively, run it on a schedule (cron or a scheduled pipeline) with a short max_hold. Each invocation then resumes and travels as far as the current timetables allow.

Upgrading from the old names

Earlier development versions used generic terms, and the project was called deploytrain. The old names are still accepted, so existing configs, scripts and state files keep working. Using an old name and its new name together is an error.

Contributing

Issues and pull requests are welcome. See CONTRIBUTING.md for the development setup, and our Code of Conduct. To report a security issue, please follow SECURITY.md rather than opening a public issue.

pip install -e ".[dev]" && python -m pytest

License

Apache License 2.0. Copyright 2026 Vishnuvardhan.