edgescaleai-cube-mcp 0.3.33.dev20260625182525

MCP server for EdgescaleAI Cube operations

cube-mcp

MCP server for EdgescaleAI Cube management and Apollo deployments.

New owner / inheriting this repo? Jump to Operations & Ownership. The two runbooks you need are docs/admin.md (full RBAC + infra reference) and docs/adding-new-user.md (onboard a teammate end-to-end). Both are written so you can point Claude Code at them and have it execute the steps for you.

Architecture

cube-agent runs locally as an MCP server (via npx cube-mcp). It handles authentication, Docker builds, Helm chart packaging, and app proxy tunnels. Server-side operations (Kubernetes, Teleport, Apollo) are proxied to cube-cloud, a FastAPI backend hosted on AWS ECS.

┌──────────────────────────────────────────────────────────────────────────────┐
│  LOCAL (developer machine)                                                   │
│                                                                              │
│  ┌─────────────┐    MCP JSON-RPC     ┌──────────────────────────────────┐   │
│  │ Claude Code  │◄──────────────────►│  cube-agent (MCP server)         │   │
│  └─────────────┘                     │                                  │   │
│                                      │  Local tools:                    │   │
│                                      │   • agent_login_browser          │   │
│                                      │   • agent_logout                 │   │
│                                      │   • agent_status                 │   │
│                                      │   • build_and_publish_to_apollo  │   │
│                                      │   • app_proxy / stop / status    │   │
│                                      │                                  │   │
│                                      │  Remote tools:                   │   │
│                                      │   (proxied to cube-cloud ──────) │   │
│                                      └────────┬──────────┬──────────────┘   │
│                                               │          │                   │
│   ~/.cube-agent/  ◄── API key storage         │          │ WebSocket         │
│   localhost:PORT  ◄── app proxy listener      │          │ /tunnel           │
│                                               │          │                   │
└───────────────────────────────────────────────┼──────────┼───────────────────┘
                                                │          │
                               HTTPS + Bearer   │          │  TCP-over-WS
                               POST /mcp/        │          │  relay
                                                │          │
┌───────────────────────────────────────────────┼──────────┼───────────────────┐
│  CLOUD (AWS ECS)                              │          │                   │
│                                               ▼          ▼                   │
│  ┌────────────────────────────────────────────────────────────────────────┐  │
│  │  cube-cloud (FastAPI)                                                  │  │
│  │                                                                        │  │
│  │  ┌─────────────────────┐     ┌──────────────────────────────────────┐ │  │
│  │  │  Auth Middleware     │     │  RBAC (Cognito + Profiles)          │ │  │
│  │  │  • Validate API key  │────►│  • Cognito group → profile          │ │  │
│  │  │    (DynamoDB lookup)│     │  • Profile → Apollo credentials     │ │  │
│  │  │  • Attach profile   │     │    (AWS Secrets Manager)            │ │  │
│  │  └─────────────────────┘     └──────────────────────────────────────┘ │  │
│  │                                                                        │  │
│  │  Cloud tools (RBAC-gated):                                            │  │
│  │  Kubernetes        Apollo Environments      Apollo Products           │  │
│  │  • cube_list       • list_environments      • list_products           │  │
│  │  • cube_status     • install_entity         • compare_product_        │  │
│  │  • kubectl_exec    • uninstall_entity         versions                │  │
│  │  • app_list        • entity_health          • list_release_channels   │  │
│  │                    • plan_details           • get_product_releases    │  │
│  │  Modules           • update_entity_config                             │  │
│  │  • list_modules    • enforce_entity_config                            │  │
│  │  • install_module                                                     │  │
│  │  • uninstall_module                                                   │  │
│  │  • update_module_  Secrets                  Registry                  │  │
│  │    variables       • create_secret          • acr_get_token           │  │
│  │                    • update_secret          • apollo_publish_manifest │  │
│  └────────────────────────┬───────────────────────────┬──────────────────┘  │
│                           │                           │                      │
│  ┌────────────────────────┴────────────────┐          │                      │
│  │  tbot sidecar (Teleport credentials)    │          │                      │
│  │  • IAM join method (Fargate identity)   │          │                      │
│  │  • identity/  → tsh commands            │          │                      │
│  │  • kube/{cluster}/ → kubeconfigs        │          │                      │
│  │  • app-{name}/ → app TLS certs          │          │                      │
│  └──────────┬─────────────────────────────┘          │                      │
│             │                                         │                      │
└─────────────┼─────────────────────────────────────────┼──────────────────────┘
              │                                         │
              │ tsh/tctl (short-lived certs)            │ GraphQL (OAuth2)
              ▼                                         ▼
┌──────────────────────────┐            ┌──────────────────────────────────┐
│  Teleport                │            │  Apollo                          │
│  edgescaleai.teleport.sh │            │  edgescaleai.palantirapollo.com  │
│                          │            │                                  │
│  • Kube clusters         │            │  • Environments & Modules        │
│  • App proxies           │            │  • Entities (Helm charts)        │
│  • SSH access            │            │  • Products & Release Channels   │
│  • Identity & TLS certs  │            │  • Change Requests               │
└──────────────────────────┘            │  • ACR (Docker + Helm registry)  │
                                        └──────────────────────────────────┘

How it works

1. Local tools run directly on the developer's machine inside cube-agent. These handle authentication (Cognito browser login, API key storage), Docker builds, Helm chart packaging, and app proxy tunnels.

2. Cloud tools are proxied from cube-agent to cube-cloud (FastAPI on AWS ECS) via MCP Streamable HTTP. Every request is authenticated with a Bearer API key validated against DynamoDB.

3. RBAC maps Cognito user groups to profiles. Each profile resolves to Apollo OAuth2 credentials stored in AWS Secrets Manager, scoping what the user can access.

4. Teleport access is provided by a tbot sidecar running alongside cube-cloud in the same ECS task. It uses IAM join to obtain short-lived certificates for Kubernetes clusters and app proxies — no tsh is needed on the developer's machine.

5. Apollo operations (environments, modules, entities, products, releases, change requests) go through a GraphQL API authenticated with per-profile OAuth2 client credentials.

Install

Requires uv (or pipx).

# Add to Claude Code
claude mcp add cube -- npx @edgescaleai/cube-mcp

Or run directly:

npx @edgescaleai/cube-mcp    # via Node (calls uvx under the hood)
uvx edgescaleai-cube-mcp     # via uv directly

Getting Started

You: "Log me in"             → agent_login_browser (opens browser for Cognito login)
You: "Show Cube status"      → cube_status
You: "What pods are failing?" → kubectl_exec (runs kubectl server-side)

Tools

Auth

Tool	Description
agent_login_browser	Log in via browser (Cognito)
agent_logout	Remove stored API key
agent_status	Check auth and connectivity, and report the cube ship flag state on this client and on cube-cloud

Kubernetes

Tool	Description
cube_list	List available Cube clusters
cube_status	Get node status for a cluster
kubectl_exec	Run kubectl commands (server-side)

Apps

Tool	Description
app_list	List Teleport apps
app_proxy	Start local proxy tunnel to an app
app_proxy_stop	Stop running proxies
app_proxy_status	Show proxy status

Build & Registry

Tool	Description
build_and_publish_to_apollo	Build Docker image, package chart, push to ACR, publish manifest
acr_get_token	Get Apollo Container Registry token
apollo_publish_manifest	Publish a manifest YAML

Apollo Environments

Tool	Description
list_environments	List/search Apollo environments
install_entity	Install a Helm chart entity
uninstall_entity	Uninstall an entity
entity_health	Get entity health and activity status
plan_details	Get plan tasks, events, and error logs
update_entity_config	Update entity config overrides
enforce_entity_config	Force re-apply entity configuration

Apollo Modules

Tool	Description
install_module	Install a module on an environment
uninstall_module	Uninstall a module
list_modules	List modules in an environment
update_module_variables	Update module variables

Apollo Secrets

Tool	Description
create_secret	Create a secret on an environment
update_secret	Update a secret value

Apollo Products & Release Channels

Tool	Description
list_products	List available products
compare_product_versions	Compare versions of a product
list_release_channels	List release channels
get_product_releases	Get releases for a product

Local Development

# One command — installs deps (incl. the editable workspace packages) and runs the suite:
make test                         # or: make test PYTEST_ARGS="packages/cube-cloud/tests -v"

# Equivalent manual steps:
uv sync --extra dev
uv pip install -e packages/cube-common -e packages/cube-cloud -e packages/cube-agent
uv run pytest packages/ -v        # bare `pytest` without the install above fails: ModuleNotFoundError: cube_cloud

# Run cube-agent locally (for debugging)
uv run cube-agent

To test with Claude Code, point the MCP server at your local code:

claude mcp add cube-local -- uv run --directory /path/to/cube-mcp cube-agent

Reload after changes with /mcp in Claude Code.

Contributing

cube-mcp uses a two-stage, deploy-on-merge model: dev-main is the integration branch and deploys to the dev environment; main is the release branch and deploys to prod. Every change flows dev-main → validated in dev → promoted to main. Don't branch off main or open feature PRs against it. A third, on-demand test stack stands apart from this promotion path — it deploys an unmerged branch for pre-merge validation and is never a merge target (see Test environment). (A few operational runbooks — e.g. the prod RBAC flow in docs/admin.md — touch prod-only resources; where the runbook itself says to, merge it directly to main and follow that runbook's steps.)

Branch and promotion flow

Branch	Cut from	Deploys to	Merge gate
feature/<desc> / fix/<desc>	dev-main	— (CI only; deployable on demand to the test stack via the deploy-test label — see Test environment)	merges into dev-main (see its gate)
dev-main	— (long-lived)	dev	PR + 1 approving review + green Run tests, branch up to date (branch-protected)
main	— (long-lived)	prod	promotion PR from dev-main + 1 approving review + green Run tests, branch up to date (branch-protected)

1. Branch from dev-main. Cut your branch off the latest dev-main and name it feature/<desc> or fix/<desc>:

git fetch origin
git switch -c feature/my-change origin/dev-main

2. Open a PR into dev-main. Make your changes, run uv run pytest packages/ -v, then open a PR targeting dev-main (not main). CI (test.yml, the Run tests check) runs automatically on every PR; dev-main is branch-protected, so merging requires an approving review and a green Run tests check with the branch up to date — enforced by GitHub for everyone, admins included. Merging deploys the change to the dev environment via deploy-dev.yml — but only for changes under its configured deploy paths (see CI/CD pipelines); docs-only changes merge without a deploy.

Optional — validate the unmerged branch on test first. For a change that touches infra or runtime behavior, exercise the branch against a real stack before it lands in dev-main: apply the deploy-test label to the PR (or run deploy-test.yml via workflow_dispatch) to deploy its unmerged head to the persistent test stack. This catches deploy- and infra-level problems without mutating dev and without merging anything. See Test environment for the trust model, the single-slot constraint, and the cube-ship resolver mode the stack runs in.

3. Validate in dev. Exercise the change against the dev stack (cube-mcp-dev-*) once it has merged to dev-main, before promoting it. Catch deploy- and infra-level problems here, not in prod.

4. Promote to main for prod. Once validated, open a promotion PR from dev-main into main (main is branch-protected: an approving review and a green Run tests check are required, branch up to date). Merge it with a merge commit — squash and rebase are disabled repo-wide, so a merge commit is the only option — so dev-main's actual commits land on main unchanged; that shared history is what keeps the two branches level (step 5). Merging deploys to prod via deploy.yml and, for changes under the publish paths, auto-publishes to PyPI + npm via publish.yml (see CI/CD pipelines). Promote dev-main as a whole rather than cherry-picking individual commits, so main only ever sees changes that were integrated and validated together.

5. Keep dev-main and main in sync. When the promotion PR is merged with a merge commit, dev-main's commits land on main unchanged and the two branches stay level — no drift. Drift appears whenever main gains a commit dev-main doesn't have: an emergency prod hotfix committed straight to main, or publish.yml's automated Release vX.Y.Z version-bump commit. (Squash- and rebase-merges — the other classic drift source, since they rewrite commits onto main — are disabled repo-wide, so a promotion can't introduce drift that way.) Whenever main moves ahead of dev-main, back-merge it so dev-main re-contains main's tip. dev-main is branch-protected with enforce_admins=true (see the table above), so the back-merge cannot be pushed straight to dev-main — route it the same way as every other change into dev-main: through a short-lived branch and a PR.

git fetch origin
git switch -c chore/back-merge-main origin/dev-main
git merge origin/main      # resolve any conflicts
git push -u origin chore/back-merge-main
# then open a PR: chore/back-merge-main -> dev-main, and merge it with a merge commit

A quick way to check for drift: git rev-list --left-right --count origin/dev-main...origin/main — the right-hand number should be 0.

See Dev environment for the full per-environment resource mapping (clusters, image tags, Terraform state).

Operations & Ownership

This section exists so anyone inheriting cube-mcp can run it end-to-end without tribal knowledge. If something here is wrong or missing, fix it in this README — don't keep the truth in your head.

Runbooks (point Claude Code at these)

Doc	Use it when
docs/admin.md	Day-to-day RBAC, profiles, API keys, Teleport, infra layout, troubleshooting. The single most important file in this repo.
docs/adding-new-user.md	Onboarding a new user (Cognito create + group assignment + verification).
docs/rbac-architecture.md	Deeper architectural background on the RBAC model.
resources/teleport.md	User-facing Teleport quickstart (tsh login, tsh kube ls, etc.).

These docs are written in runbook style — open Claude Code in this repo and ask it to "follow docs/adding-new-user.md to add [e-mail] to the lear-dev group" and it will execute the AWS CLI calls itself.

Where AWS resources live

Everything is in AWS account 992382448282, region us-west-2. All resources are managed by Terraform in infra/terraform/ — change them there, not in the console.

Resource	Name / ARN suffix	Terraform file
Cognito user pool	cube-mcp-prod-* (look up with aws cognito-idp list-user-pools)	infra/terraform/cognito.tf
Cognito groups	admin, lear-dev, conagra-dev, pltr-dev, edgescaleai-dev	infra/terraform/cognito.tf
API keys table (DynamoDB)	cube-mcp-prod-api-keys	infra/terraform/dynamodb.tf
Profile credentials (Secrets Manager)	cube-mcp/profiles/<profile>	infra/terraform/secrets.tf
Prod-approver identity (Secrets Manager)	cube-mcp/profiles/prod-approver — the cube ship prod-approval module's server-held Apollo identity (SCRUM-1063); the sole identity cube-cloud places to prod under. A prod ship default-denies until it is populated.	infra/terraform/secrets.tf
tbot config (Secrets Manager)	cube-mcp/tbot-config	infra/terraform/tbot.tf
Inventory read token (Secrets Manager)	cube-mcp/inventory/read-token	infra/terraform/secrets.tf
ECS task role	cube-mcp-prod-ecs-task	infra/terraform/iam.tf
ECS cluster	cube-mcp-prod-cluster (service: cube-mcp-prod-service)	infra/terraform/ecs.tf
CloudWatch Logs	/ecs/cube-mcp-prod (cube-cloud + tbot containers)	infra/terraform/ecs.tf
ALB + HTTPS listener	cube.edgescaleai-cube.com	infra/terraform/alb.tf
Route53 records	edgescaleai-cube.com zone (Z0571857327BEMX2EHNZU)	infra/terraform/route53.tf
ECR (cube-cloud + tbot images)	cube-mcp/cube-cloud, cube-mcp/tbot	infra/terraform/ecr.tf, tbot.tf
SES sender	[e-mail] (identity edgescaleai-cube.com)	infra/terraform/cognito.tf

Outside AWS:

• Teleport: edgescaleai.teleport.sh — bot is cube-mcp-bot, joins via IAM. See docs/admin.md §4.
• Apollo: edgescaleai.palantirapollo.com — OAuth2 credentials per profile, stored in Secrets Manager.
• PyPI / npm: auto-published from main via CI (GitHub Actions). The npm shim under npm/ is a thin wrapper that shells to uvx edgescaleai-cube-mcp.

CI/CD pipelines

All deploys go through GitHub Actions in .github/workflows/. AWS auth uses OIDC — no long-lived AWS keys are stored in GitHub.

Workflow	Trigger	What it does
test.yml	Reusable, called by others	Runs pytest packages/ -v.
deploy.yml	Push to main (paths: packages/cube-cloud, cube-common, src/, infra/)	Runs tests → terraform apply (only if infra/terraform/ changed) → builds & pushes cube-mcp/cube-cloud:latest and cube-mcp/tbot:latest to ECR → seeds new profile secrets with admin creds → pushes infra/tbot/tbot-config-prod.yaml to Secrets Manager → aws ecs update-service --force-new-deployment and waits for services-stable. Also posts terraform plan as a PR comment for terraform PRs.
deploy-dev.yml	Push to dev-main branch	Same shape as deploy.yml, but targets cube-mcp-dev-cluster / cube-mcp-dev-service and uses TF state key cube-cloud-dev/terraform.tfstate. Images tagged :dev. Use this branch to test infra changes before merging to main.
deploy-test.yml	On demand — workflow_dispatch (pick any branch) or a deploy-test label on a PR (deploys the PR head)	Deploys an unmerged feature branch to the persistent test stack (cube-mcp-test-cluster / cube-mcp-test-service, root config infra/terraform/envs/test/, state cube-mcp/test.tfstate, images tagged :test) without touching dev or prod. Runs tests → builds & pushes :test images → writes a provider_override.tf (drops the baked profile for OIDC) → terraform init -backend-config="profile=" → terraform apply → --force-new-deployment → surfaces the deployed URL in the run summary and (for the label path) as a PR comment. Single concurrent slot. See Test environment.
publish.yml	Push to main (paths: src/, npm/, pyproject.toml)	Runs tests → bumps the patch version in pyproject.toml and npm/package.json → commits as Release vX.Y.Z [skip ci] and tags vX.Y.Z → publishes to PyPI (twine) and npm (npm publish --access public).
sync-knowledge.yml	Scheduled / manual	Syncs internal Claude Code knowledge from the disco-projects repo. Not on the critical deploy path — safe to ignore unless it breaks.

OIDC role for all AWS-touching jobs: arn:aws:iam::992382448282:role/github-actions.

Required CI secrets

Stored as GitHub repo secrets. To rotate any of these, generate a new value at the source and update via gh secret set <NAME> (or the repo Settings → Secrets UI).

Secret	Where used	Source / how to rotate
PYPI_TOKEN	publish.yml	pypi.org/manage/account/token/ — scope to the cube-mcp project.
NPM_TOKEN	publish.yml	npm account → Access Tokens → Automation token (must allow publish).
PUBLISH_DEPLOY_KEY	publish.yml	SSH deploy key on this repo with write access (used to push the auto-bumped version commit + tag back to main). Regenerate: create a new SSH keypair, add the public key as a repo Deploy Key with write access, paste the private key into the secret.
DISCO_PROJECTS_TOKEN	sync-knowledge.yml	Fine-grained PAT with read access to EdgescaleAI/disco-projects.
ANTHROPIC_API_KEY	sync-knowledge.yml	console.anthropic.com — used by the disco knowledge sync job.

AWS credentials are not stored as secrets — deploy.yml/deploy-dev.yml assume arn:aws:iam::992382448282:role/github-actions via OIDC. To grant a new repo or change permissions, update that role's trust policy and inline policies in AWS IAM.

Terraform state

Stored in S3, no DynamoDB lock table currently configured (single-writer assumption — CI is the only applier).

Env	Bucket	Key
Prod	cube-mcp-terraform-state (us-west-2)	cube-cloud/terraform.tfstate
Dev	cube-mcp-terraform-state (us-west-2)	cube-cloud-dev/terraform.tfstate
Test	esai-dev-terraform-state (us-west-2)	cube-mcp/test.tfstate

Prod and dev run from the flat root config at infra/terraform/ (selected by -backend-config key + -var-file). Test runs from its own convention-aligned root config at infra/terraform/envs/test/ — own backend (org bucket, encrypt + use_lockfile, baked esai-dev profile), state in esai-dev-terraform-state. dev/prod migrate onto the same envs//modules/ shape later under SCRUM-1151.

Local apply (only if you really need to bypass CI):

# prod / dev — flat config
cd infra/terraform
terraform init                                                          # prod
terraform init -backend-config="key=cube-cloud-dev/terraform.tfstate"   # dev
terraform plan

# test — own root config (backend baked in versions.tf)
cd infra/terraform/envs/test
terraform init
terraform plan

The S3 bucket has versioning enabled, so a corrupt state file can be rolled back to a previous version via the AWS console.

Dev environment

Parallel stacks exist in the same AWS account for testing changes before merging to main.

	Prod	Dev	Test
Trigger	push to main	push to dev-main	on demand (workflow_dispatch / deploy-test PR label)
ECS cluster	cube-mcp-prod-cluster	cube-mcp-dev-cluster	cube-mcp-test-cluster
ECS service	cube-mcp-prod-service	cube-mcp-dev-service	cube-mcp-test-service
Image tag	:latest	:dev	:test
TF state key	cube-cloud/terraform.tfstate	cube-cloud-dev/terraform.tfstate	cube-mcp/test.tfstate (in esai-dev-terraform-state)
TF root / vars	infra/terraform/ (defaults)	infra/terraform/ + environments/dev.tfvars	infra/terraform/envs/test/ (inputs baked in main.tf)
Domain / TLS	cube.edgescaleai-cube.com (ACM + Route53)	cube-dev.edgescaleai-cube.com	none — HTTP on the raw ALB DNS

Push to dev-main to validate, then open a promotion PR dev-main → main for prod. See Contributing for the full branch/promotion flow and how to keep the two branches from drifting.

Test environment

A third persistent stack (cube-mcp-test-*) for validating an unmerged feature branch against a real environment without disturbing dev or prod (SCRUM-996). Unlike dev and prod it is not wired to a branch — deploy-test.yml deploys on demand. The change you are validating is never merged; only the deploy tooling itself lives on a branch (see One-time setup below).

Identifiers (also in the env table above): cluster cube-mcp-test-cluster, service cube-mcp-test-service, image tag :test, root config infra/terraform/envs/test/, state cube-mcp/test.tfstate in esai-dev-terraform-state. The app is served over plain HTTP at the ALB DNS name (no domain), e.g. http://cube-mcp-test-alb-<id>.us-west-2.elb.amazonaws.com.

How to deploy to it

Two triggers, with different reach:

Trigger	What it deploys	Where the workflow must exist	Use it when
deploy-test label on a PR	the PR's head commit (unmerged)	in the PR itself — pull_request runs the workflow from the PR's merge ref, so any branch cut from dev-main (after this lands there) has it automatically	the normal case — you have a PR open and want to see it run live
workflow_dispatch	any branch you pick in the Actions UI	the default branch (main)	a branch with no open PR, or a one-off

The label path is sufficient for the core use case and is all you need day to day — workflow_dispatch is a convenience that only becomes available once the workflow reaches main via the normal dev-main → main promotion (GitHub only exposes workflow_dispatch for workflows on the default branch). After deploy, the URL, the deployed commit, and the MCP endpoint are written to the run summary, and (label path only) posted as a PR comment.

Trust model. The deploy-test label is the trigger gate, and GitHub lets any collaborator with triage rights (not just write) apply a label — so labeling a contributor's PR sends its unmerged head commit to AWS under the github-actions OIDC role. This is deliberate, not an oversight: the blast radius is bounded by design — the role can update the existing test stack but not create one, the test environment owns no shared resources (see Shared resources), and the :test image tag is distinct from :dev / :latest. Treat applying the label as granting a deploy accordingly.

Redeploying / resetting the slot. There is a single concurrent slot (desired_count = 1), serialized by the workflow's concurrency: deploy-test group, so one branch is under test at a time. To redeploy: re-run the Deploy Test workflow, or remove and re-add the deploy-test label (GitHub only fires on the labeled transition, so re-applying an already-present label does nothing). To wipe state entirely, terraform destroy (below) and redeploy.

Cheap, isolated tier

• domain = "" → plain HTTP on the ALB DNS, no ACM cert and no Route53 record (alb.tf creates the HTTP-forward listener instead of HTTPS).
• The tbot sidecar is non-essential (essential = false); the app runs without Teleport credentials (/health reports tbot.ready: false, which is expected here).
• The cube ship inventory resolver runs in stub mode (cube_ship_inventory_mode = "stub") — canned data, no credentials, and no dependency on the dev-owned cube-mcp/inventory/read-token secret. Real mode (against the dev-testing edgescale-inventory tier) is deferred until that shared token is restored on the dev side; flip cube_ship_inventory_mode = "real" + set inventory_api_url in envs/test/main.tf then.
• cube ship's prod-promotion leg reads an isolated Engine M2M client: engine_secret_id = "cube-mcp/engine-client/test" (set in envs/test/main.tf), not the account-global cube-mcp/engine-client the dev/prod stacks share, so a test run never authenticates to Engine as the dev/prod identity. The secret is populated out of band and covered by the task role's cube-mcp/engine-client* wildcard grant; engine_base_url keeps its engine-dev default. Until the secret is populated the prod hop default-denies (there is no direct-Apollo fallback).

Authentication (important)

The test stack has its own isolated Cognito user pool (cube-mcp-test-users), API-key table (cube-mcp-test-api-keys), and login-session table — nothing carries over from dev or prod, and the pool starts with zero users. Because domain = "", the Cognito hosted-UI / browser login is disabled (agent_login_browser will not work here). Authenticate instead via:

• API key — the MCP endpoint expects Authorization: Bearer <api-key> (a request without it returns 401). Mint a key against the test stack (see docs/admin.md); keys minted for dev/prod do not work here.
• Programmatic Cognito — the app client enables ALLOW_USER_PASSWORD_AUTH and ALLOW_USER_SRP_AUTH, so a user created in the cube-mcp-test-users pool can authenticate with username/password (no hosted UI required) to bootstrap a key.

Shared resources

The account-global, fixed-name resources every environment's Terraform would otherwise declare — the ECR repos (cube-mcp/cube-cloud, cube-mcp/tbot), the cube-mcp/profiles/* secrets, cube-mcp/tbot-config, and cube-mcp/inventory/read-token — are referenced here rather than recreated, gated by manage_shared_resources = false in infra/terraform/envs/test/main.tf. The ECR repo URLs are derived (<account>.dkr.ecr.<region>.amazonaws.com/<repo>) rather than read via an aws_ecr_repository data source — that data source calls ecr:DescribeImages, which the github-actions deploy role isn't granted; the inventory-token ARN (which has a random suffix) is read via a data source only in real inventory mode (covered by the role's secretsmanager:DescribeSecret); in stub mode — the test default — it is not read, so the stack carries no dependency on that shared secret. Prod and dev keep ownership (the variable defaults to true, and moved blocks migrate their existing state to the count-indexed addresses in place — no destroy/recreate). The :test image is pushed to the shared cube-mcp/cube-cloud and cube-mcp/tbot repos, just under a distinct tag. The one secret the test stack deliberately does not share is the Engine M2M client: it reads its own cube-mcp/engine-client/test (engine_secret_id in envs/test/main.tf), so the prod-promotion leg never presents the dev/prod Engine identity. That entry is populated out of band and is not declared by any stack's Terraform (the shared cube-mcp/engine-client shell is created only by manage_shared_resources envs).

One-time setup (do this once)

1. Land the tooling. Merge this change to dev-main to enable the deploy-test label path. (It rides to main on the next routine promotion, which additionally enables workflow_dispatch.) Make sure the repo has a deploy-test label.

2. Grant the OIDC role org-bucket access (admin, one-time). test state lives in esai-dev-terraform-state (the org backend bucket), not cube-mcp's own bucket. The github-actions OIDC role must be able to read/write it under the cube-mcp/* key prefix (incl. the S3-native lock object): s3:GetObject/PutObject/DeleteObject on arn:aws:s3:::esai-dev-terraform-state/cube-mcp/* and s3:ListBucket on the bucket. The role is org-managed (not defined in this repo), so the grant lives wherever that role's policy lives. Needed for ongoing CI; the admin bootstrap below uses your own credentials and does not depend on it.

3. Bootstrap the stack (admin). The github-actions OIDC role can operate an existing stack (image rebuild + ecs:UpdateService redeploy, in-place Terraform updates) but not create one from scratch — it lacks ecs:CreateCluster/CreateService, elasticloadbalancing:CreateLoadBalancer/CreateTargetGroup, cognito-idp:CreateUserPool*, iam:CreateRole, etc. So, exactly like dev and prod, the test stack must be stood up once by an admin running Terraform locally:

   aws sso login --profile esai-dev
   cd infra/terraform/envs/test
   terraform init
   terraform apply   # needs admin creds; one time
   

   After that, deploy-test.yml handles ongoing deploys. App-only feature branches (the common case) just rebuild the :test image and redeploy, which the OIDC role can do; a branch that adds new infra may need its resources created by an admin apply first (or the OIDC role's policy extended with the create actions).

Cost & teardown

The stack runs an ALB plus one Fargate task continuously. It is persistent by design, but to drop it to zero cost between uses:

aws sso login --profile esai-dev
cd infra/terraform/envs/test
terraform init
terraform destroy

Because this environment owns no shared resources — their URLs are derived, the inventory-token ARN is only ever read via a data source (and in the default stub mode not read at all), and the profile/tbot secrets aren't declared at all — destroy only removes the cube-mcp-test-* resources; the ECR repos, profile secrets, tbot config, and inventory token (owned by prod/dev) are left untouched. Redeploy any time via the label or workflow_dispatch.

Per-PR ephemeral environments were evaluated and deferred as higher-effort (recurring shared-resource handling and create/destroy lifecycle automation on every apply); the single persistent slot is the pragmatic first step and can graduate to ephemeral if concurrent multi-branch testing becomes the bottleneck. See SCRUM-996.

Versioning & releases

• Scheme: patch-bump only, automated. publish.yml reads pyproject.toml, increments the patch, writes it back to both pyproject.toml and npm/package.json, commits as github-actions[bot], and tags vX.Y.Z.
• Major/minor bumps: edit pyproject.toml manually in a PR. The next merge to main that touches src/ / npm/ / pyproject.toml will publish from there.
• PyPI ↔ npm sync: the same version number is used for both packages. The npm/ shim is a thin wrapper that shells to uvx edgescaleai-cube-mcp.
• What "main" means for users: every merge that touches the publish paths ships to PyPI + npm within minutes. There is no staging release.

Rollback playbook

Surface	How to roll back
ECS deploy (cube-cloud or tbot)	Images are tagged :latest only — there is no previous-image tag to roll forward to. Rollback path: git revert the offending commit on main; the next deploy.yml run will rebuild and redeploy. For an emergency, manually re-tag a known-good ECR image as :latest (aws ecr batch-get-image → put-image) and aws ecs update-service --force-new-deployment. Improvement worth making: tag images with the git SHA so rollback is one CLI call.
Terraform	git revert the offending commit; deploy.yml re-runs terraform apply on next push. For state corruption, restore from S3 versioning on cube-mcp-terraform-state.
PyPI / npm	Cannot un-publish (yanking PyPI is reversible only in narrow windows; npm allows unpublish within 72h). The standard path is to push a new patch version with the fix. Users on npx @edgescaleai/cube-mcp and uvx edgescaleai-cube-mcp pick up the new version on next invocation.
Cognito user / group change	Cognito has no native rollback. Reverse the change manually: re-add the user, restore group membership. Audit trail is in CloudTrail (cube-mcp-prod-trail if enabled — verify).
Secrets Manager profile credentials	Each secret has versioning. Roll back with aws secretsmanager update-secret-version-stage --secret-id cube-mcp/profiles/<profile> --version-stage AWSCURRENT --move-to <prev_version_id>.

Monitoring & alerting

Status: not yet configured. No CloudWatch alarms, SNS topics, Sentry, or paging are set up at the time of this handoff. The only observability is CloudWatch Logs (/ecs/cube-mcp-prod) and the ECS service's own task health.

Day-to-day debugging:

# Live tail prod logs (cube-cloud + tbot)
aws logs tail /ecs/cube-mcp-prod --follow --region us-west-2

# Filter to just tbot
aws logs tail /ecs/cube-mcp-prod --filter-pattern "tbot" --follow

# Service health
aws ecs describe-services \
  --cluster cube-mcp-prod-cluster \
  --services cube-mcp-prod-service \
  --query 'services[0].{running:runningCount,desired:desiredCount,events:events[0:5]}'

Recommended additions for the next owner (none of these exist yet):

• CloudWatch alarm on ECS service RunningTaskCount < 1 → SNS → email/PagerDuty.
• ALB 5xx alarm on the listener.
• DynamoDB throttling alarm on cube-mcp-prod-api-keys.
• Synthetic check that runs agent_login_browser end-to-end weekly.

Stakeholders & contacts

TODO(owner): Vinayak (departing) is the only person with full context on these. Fill these in before he leaves so the next owner has someone to call.

Role	Person	Contact
Internal product owner	TODO
Eng escalation / on-call	TODO
AWS account admin	TODO
Teleport admin (can grant tctl access)	TODO
Apollo admin (can issue per-tenant OAuth2 creds)	TODO
Lear tenant POC	TODO
Conagra tenant POC	TODO
Palantir tenant POC	TODO
EdgescaleAI internal dev (edgescaleai-dev profile)	TODO

Cost & billing

TODO(owner): Add rough monthly run-rate and the account that gets billed.

• AWS (account 992382448282): TODO — typical monthly spend, biggest line items, who pays the invoice.
• Teleport (edgescaleai.teleport.sh): TODO — plan tier, seat count, billing contact.
• Apollo (edgescaleai.palantirapollo.com): TODO — Palantir contract reference, who renews.
• Domain (edgescaleai-cube.com): TODO — registrar, expiry date, who has the login.
• PyPI / npm: free tiers, no recurring cost.

If costs need to be cut quickly: the dev ECS service (cube-mcp-dev-service) can be scaled to zero with no user impact.

Access & disaster recovery

Surface	Status / what the next owner should verify
GitHub repo admins	TODO(owner): list everyone with admin/maintain on EdgescaleAI/cube-mcp and confirm branch protection on main (require PR + passing tests).
AWS root account access	TODO(owner): confirm who holds the root credentials for account 992382448282 and that MFA is enforced.
Domain registrar login	TODO(owner): who has the registrar account for edgescaleai-cube.com.
DynamoDB cube-mcp-prod-api-keys	Point-in-time recovery enabled (35-day window).
Secrets Manager	Per-secret versioning enabled by default — see Rollback playbook.
Terraform state	S3 bucket versioning enabled on cube-mcp-terraform-state.
ECR images	Lifecycle policy TODO: verify untagged-image cleanup so cost doesn't grow unbounded.

Known tech debt / planned work

Update this list as you ship or de-scope items so the next owner sees current state, not folklore.

• Phase 2 RBAC: per-role tbot identity outputs are wired up but only admin profile actually has scoped Apollo creds. Lear/Conagra/Palantir profiles still resolve to admin Apollo credentials at runtime — see the note in docs/admin.md §3 ("Currently defined groups"). Scoping these is the biggest open item.
• ECR :latest-only tagging makes ECS rollback awkward (see Rollback playbook). Tag images with the git SHA in deploy.yml.
• No alerting. See Monitoring section.
• Teleport role management is manual tctl. The future plan is the Teleport Terraform provider — see docs/admin.md → Future.
• No DynamoDB lock on Terraform state. Acceptable while CI is the only applier; add a lock table if local applies become routine.
• Cognito drift: esaiadmin group. Surfaced in #93. The prod Cognito pool has an esaiadmin group that isn't defined in infra/terraform/cognito.tf — it was created in the console. Either terraform import it or delete it; don't leave it as drift.
• Stale claim in docs/rbac-architecture.md line ~134 that "Cognito user → profile assignment is not yet implemented." This contradicts docs/admin.md and login_flow.py (the groups[0] resolution has shipped). Fix in a docs-only PR — also flagged in #93.
• TODO(owner): anything else in flight at handoff time.

First-day checklist for a new owner

1. Read docs/admin.md end-to-end. Then run agent_login_browser yourself so you've experienced the user flow.
2. Get console access to AWS account 992382448282 (us-west-2) with permissions for Cognito, DynamoDB, Secrets Manager, ECS, IAM, and CloudWatch Logs.
3. Get Teleport admin (tctl) on edgescaleai.teleport.sh — needed to manage the cube-mcp-bot and add roles. Ask the person listed in Stakeholders & contacts.
4. Get Apollo admin on edgescaleai.palantirapollo.com — needed to issue per-profile OAuth2 credentials when adding new tenants.
5. Install cube-admin locally for API key management: uv pip install -e packages/cube-admin. This package is internal-only and must never be published.
6. Verify CI is healthy and you can rotate every secret in Required CI secrets. Push a no-op commit to dev-main to confirm deploy-dev.yml succeeds end-to-end against the dev stack.
7. Replace every TODO(owner) in this README — Stakeholders, Cost, Access & DR, Tech debt. If you can't fill one in, write down who you asked and what they said. Don't leave them blank.

Common ops tasks

• Add a user: docs/adding-new-user.md.
• Add a new tenant / role: docs/admin.md → Creating a new role / profile. Six steps — Cognito group, SM secret, Teleport role, bot role grant, tbot output, merge.
• Revoke access: docs/admin.md → Revoking access.
• Manage API keys: cube-admin keys {create,list,revoke} — see docs/admin.md → Managing API keys with cube-admin.
• Debug a failing tool call: check aws logs tail /ecs/cube-mcp-prod --follow for the cube-cloud and tbot containers.

Repo layout cheatsheet

packages/
  cube-agent/    # Local MCP server (npx cube-mcp entrypoint)
  cube-cloud/    # FastAPI service on ECS — auth, RBAC, Apollo, Teleport relay
  cube-common/   # Shared types and helpers
  cube-admin/    # Internal-only API key CLI — DO NOT PUBLISH
infra/
  terraform/     # All AWS resources
  tbot/          # tbot Dockerfile, entrypoint, prod config
docs/            # Runbooks (start here)
resources/       # User-facing reference docs
npm/             # npm shim that calls uvx edgescaleai-cube-mcp
scripts/         # Release / publish helpers