cube-mcp

MCP server for EdgescaleAI Cube management and Apollo deployments.

New owner / inheriting this repo? Jump to Operations & Ownership. The two runbooks you need are docs/admin.md (full RBAC + infra reference) and docs/adding-new-user.md (onboard a teammate end-to-end). Both are written so you can point Claude Code at them and have it execute the steps for you.

Architecture

cube-agent runs locally as an MCP server (via npx @edgescaleai/cube-mcp). It handles authentication, Docker builds, Helm chart packaging, and app proxy tunnels. Server-side operations (Kubernetes, Teleport, Apollo, Groundcover observability) are proxied to cube-cloud, a FastAPI backend hosted on AWS ECS.

┌──────────────────────────────────────────────────────────────────────────────┐
│  LOCAL (developer machine)                                                   │
│                                                                              │
│  ┌─────────────┐    MCP JSON-RPC     ┌──────────────────────────────────┐   │
│  │ Claude Code  │◄──────────────────►│  cube-agent (MCP server)         │   │
│  └─────────────┘                     │                                  │   │
│                                      │  Local tools:                    │   │
│                                      │   • agent_login_browser          │   │
│                                      │   • agent_logout                 │   │
│                                      │   • agent_status                 │   │
│                                      │   • build_and_publish_to_apollo  │   │
│                                      │   • app_proxy / stop / status    │   │
│                                      │                                  │   │
│                                      │  Remote tools:                   │   │
│                                      │   (proxied to cube-cloud ──────) │   │
│                                      └────────┬──────────┬──────────────┘   │
│                                               │          │                   │
│   ~/.cube-agent/  ◄── API key storage         │          │ WebSocket         │
│   localhost:PORT  ◄── app proxy listener      │          │ /tunnel           │
│                                               │          │                   │
└───────────────────────────────────────────────┼──────────┼───────────────────┘
                                                │          │
                               HTTPS + Bearer   │          │  TCP-over-WS
                               POST /mcp/        │          │  relay
                                                │          │
┌───────────────────────────────────────────────┼──────────┼───────────────────┐
│  CLOUD (AWS ECS)                              │          │                   │
│                                               ▼          ▼                   │
│  ┌────────────────────────────────────────────────────────────────────────┐  │
│  │  cube-cloud (FastAPI)                                                  │  │
│  │                                                                        │  │
│  │  ┌─────────────────────┐     ┌──────────────────────────────────────┐ │  │
│  │  │  Auth Middleware     │     │  RBAC (Cognito + Profiles)          │ │  │
│  │  │  • Validate API key  │────►│  • Cognito group → profile          │ │  │
│  │  │    (DynamoDB lookup)│     │  • Profile → Apollo credentials     │ │  │
│  │  │  • Attach profile   │     │    (AWS Secrets Manager)            │ │  │
│  │  └─────────────────────┘     └──────────────────────────────────────┘ │  │
│  │                                                                        │  │
│  │  Cloud tools (RBAC-gated):                                            │  │
│  │  Kubernetes        Apollo Environments      Apollo Products           │  │
│  │  • cube_list       • list_environments      • list_products           │  │
│  │  • cube_status     • install_entity         • compare_product_        │  │
│  │  • kubectl_exec    • uninstall_entity         versions                │  │
│  │  • app_list        • entity_health          • list_release_channels   │  │
│  │  • list_services   • plan_details           • get_product_releases    │  │
│  │                    • cancel_active_plan                               │  │
│  │  Expose (Teleport) • update_entity_config   Groundcover (read-only)   │  │
│  │  • expose          • enforce_entity_config  • groundcover_list_*      │  │
│  │  • unexpose                                 • groundcover_query_*     │  │
│  │  • list_exposed    Modules                  • groundcover_search_*    │  │
│  │                    • list_modules           • groundcover_k8s_events  │  │
│  │  Secrets           • install_module                                   │  │
│  │  • create_secret   • uninstall_module       Registry                  │  │
│  │  • update_secret   • update_module_         • acr_get_token           │  │
│  │                      variables                                        │  │
│  │                                             Feedback                  │  │
│  │                                             • send_feedback         │  │
│  └────────────────────────┬───────────────────────────┬──────────────────┘  │
│                           │                           │                      │
│  ┌────────────────────────┴────────────────┐          │                      │
│  │  tbot sidecar (Teleport credentials)    │          │                      │
│  │  • IAM join method (Fargate identity)   │          │                      │
│  │  • identity/  → tsh commands            │          │                      │
│  │  • kube/{cluster}/ → kubeconfigs        │          │                      │
│  │  • app-{name}/ → app TLS certs          │          │                      │
│  └──────────┬─────────────────────────────┘          │                      │
│             │                                         │                      │
└─────────────┼─────────────────────────────────────────┼──────────────────────┘
              │                                         │
              │ tsh/tctl (short-lived certs)            │ GraphQL (OAuth2)
              ▼                                         ▼
┌──────────────────────────┐            ┌──────────────────────────────────┐
│  Teleport                │            │  Apollo                          │
│  edgescaleai.teleport.sh │            │  edgescaleai.palantirapollo.com  │
│                          │            │                                  │
│  • Kube clusters         │            │  • Environments & Modules        │
│  • App proxies           │            │  • Entities (Helm charts)        │
│  • SSH access            │            │  • Products & Release Channels   │
│  • Identity & TLS certs  │            │  • Change Requests               │
└──────────────────────────┘            │  • ACR (Docker + Helm registry)  │
                                        └──────────────────────────────────┘

                        │ HTTPS (per-cube, read-only)
                        ▼
            ┌──────────────────────────────────┐
            │  Groundcover                     │
            │  api.groundcover.com             │
            │  • Metrics / logs / traces       │
            │  • K8s events & workload health  │
            └──────────────────────────────────┘

A separate browser front door (not shown above) serves `expose`d in-cube web
apps to account-less external users: browser → `<app>-<env>` wildcard subdomain
→ ALB → cube-cloud browser proxy → tbot mTLS dial into the cube. See
[docs/browser-app-proxy-spec.md](docs/browser-app-proxy-spec.md).

How it works

1. Local tools run directly on the developer's machine inside cube-agent. These handle authentication (Cognito browser login, API key storage), Docker builds, Helm chart packaging, and app proxy tunnels.

   The sign-in page also offers self-service password reset — "Forgot your password?" sends a code to the user's email (Cognito ForgotPassword), and redeeming it signs them straight in, so no operator conveys a temporary password by hand. The request step answers identically whether or not the address has an account: Cognito distinguishes "no such user", "no verified email" and "still on a temporary password", and all three are collapsed into one page so the flow cannot be used to enumerate accounts. It covers CONFIRMED users only — Cognito refuses ForgotPassword for a user who never completed first login or whose invite expired, so those are re-invited by an operator with scripts/reset-password.py --resend-invite, which has Cognito email the new password directly. See docs/admin.md §6.

2. Cloud tools are proxied from cube-agent to cube-cloud (FastAPI on AWS ECS) via MCP Streamable HTTP. The MCP endpoint (/mcp) and the tunnel relay (/tunnel) are authenticated with a Bearer API key validated against DynamoDB. (The browser surfaces additionally accept the same key from a parent-domain session cookie — on app subdomains, and on the apex under the /console prefix only; /mcp and /tunnel stay Bearer-only as a CSRF gate.)

3. RBAC maps Cognito user groups to profiles. Each profile resolves to Apollo OAuth2 credentials stored in AWS Secrets Manager, scoping what the user can access. The same secret carries the profile's app companies — the set of company labels whose exposed apps it can see, home first — which scripts/provision-access.py writes together with the Teleport role (ADR-0002). A stack reads that set only when its Terraform app_companies_source is stored (dev); prod stays on map, the in-code baseline, until the multi-company transition (spec) — so until then a profile whose HOME app company differs from its map entry also needs that entry deployed (the CLI warns).

4. Teleport access is provided by a tbot sidecar running alongside cube-cloud in the same ECS task. It uses IAM join to obtain short-lived certificates for Kubernetes clusters and app proxies — no tsh is needed on the developer's machine. The console's Teleport connectivity chip reads the same sidecar's tsh kube ls — one fleet-wide, cached call — and reports unknown rather than "disconnected" whenever it cannot ask.

5. Apollo operations (environments, modules, entities, products, releases, change requests) go through a GraphQL API authenticated with per-profile OAuth2 client credentials.

6. Groundcover provides read-only, per-cube observability (metrics, logs, traces, K8s events, workload health) via the groundcover_* tools. Access is scoped to the cubes your Apollo credentials can see, and every query is pinned server-side to a single cube — see docs/groundcover-integration-spec.md.

7. Feedback has two destinations behind one tool, send_feedback. A bug report — the wire value is still support — files a ticket in Freshservice (freshservice_enabled). It is presented as a bug report rather than a support request on purpose: the integration is complete, the on-call rotation and SLAs behind the word "support" are not, and the wording is held back until they are (console-spec §4.4e). A feature request raises a customer request on the Jira Service Management Customer Feedback board (CF), which is a product backlog (jira_enabled). Both flags default off per stack, and both credentials are service accounts holding a single write scope. The reporter is resolved from the caller's own API key rather than from a tool argument, on both paths.

8. Browser app proxy gives account-less external users a browser-only path to in-cube web apps you've exposed — no Teleport account and no local agent. Requests hit an <app>-<env> wildcard subdomain, authenticate via a Cognito session cookie, and are dialed into the cube over the same tbot mTLS path. This is the URL every surface publishes — expose, list_exposed, app_list and the console all build it through one function (browser_proxy/resolver.py::browser_app_url), which validates the host against the same rule the proxy parses it back with, so a published URL is always one the proxy will route. See docs/browser-app-proxy-spec.md.

9. Console is the browser UI at https://<apex>/console — sign in at the apex and you land on it. A React SPA (built from console/web, compiled by a node stage in the cube-cloud Dockerfile) over JSON endpoints that share the Apollo query + RBAC layer with the MCP tools. Cookie-authenticated, bookmarkable, and deliberately narrow: reads are GET-only and same-origin-enforced, and there is no generic tool-proxy endpoint. Each app card links the app's public browser-app-proxy URL (https://<app>-<cube>.<apex> — not Teleport's …teleport.sh address, which needs a Teleport account), and offers an inline Unexpose behind a confirmation that names the cube. Each app also links a Logs view at its own bookmarkable URL (/console/cubes/<cube>/apps/<app>/logs), backed by Groundcover — one query per Kubernetes workload the app owns, merged newest-first, and near-live: it polls every 10s incrementally (only rows newer than the last window), pauses when the tab is hidden, and offers a Live/Paused toggle. Streaming was deliberately not used — Groundcover exposes only search endpoints, so a WebSocket would still poll underneath while adding a second auth path. Each app row also offers Expose, which lists the cube's Kubernetes Services as candidates (auto-selecting the port when a Service has exactly one, and requiring a choice otherwise — a wrong port surfaces later as a Teleport timeout) and refuses any target the console did not itself enumerate. Expose and unexpose are the console's only mutations: it is its own route with enumerated args, gated on origin + a X-Console-CSRF synchronizer token, and it fails closed where the CSRF secret is unprovisioned. The topbar carries a Dark / Light theme toggle (dark by default, remembered per browser; the guide has the same toggle and receives the choice as ?theme=light, since it stores nothing — see docs/console-spec.md §4.11). The topbar links the Cube Field Guide at /console/guide — a 21-lesson onboarding course (Quick Start / Deep Dive / Appendix). It lives in the repo at console/web/public/guide/ (index.html + guide.css + guide.js), which is its source of truth: edit it there. It was imported once from an authored single-file HTML by scripts/vendor-field-guide.py — a one-shot, in no build path, which also stripped the guide's progress-tracking state and its residual internal content. Those removals are now invariants, enforced on every build by console/web/src/guide.test.ts (which renders every lesson in jsdom and sweeps for state, network calls, and customer names). It is a separate static document with its own, deliberately separate, CSP profile; read docs/console-spec.md §4.11 before editing the guide or either CSP. Cube cards and the cube page show Apollo's display name where it has one, plus a rolled-up agent status chip (Connected / Stale / Never reported / Mixed) — mixed is a real state, so it is surfaced rather than resolved to whichever agent Apollo listed first. Each app row shows when it last changed — the newest successful Apollo plan, so a failing retry loop doesn't read as a change; where the plan page's ordering can't be confirmed it shows no date rather than a possibly-stale one (§4.15). The cube page carries an Online / Offline / Unknown badge beside the cube name, derived from how recently Groundcover holds a log for that cube — hovering gives "Last seen X ago" and names the source. Offline renders grey, not red — a cube may be air-gapped or on a network that forbids egress to Edgeport, which is intended operation rather than a fault, and the tooltip says so. Unknown is a real state and never renders as offline: a cube Groundcover has never heard of, or a failed query, says so rather than claiming the cube is down. See docs/console-spec.md §4.14. Beside it sits a Teleport chip (Connected / Disconnected / Not enrolled / Unknown) — the third reachability signal, and the one that answers can the platform reach into this cube, since kubectl_exec, the service list, expose and the app proxy all ride the Teleport agent. Each chip names the system it measures, so three of them reading "Offline / Stale / Connected" is information rather than a contradiction. It joins on the cube's teleport-agent-module resourceId — a UUID minted by the module install, matched against Teleport's teleport.internal/resource-id label. Not the cluster name, which is caller-writable and so made this read a cross-tenant liveness oracle; and not the environment name, which diverges on three cubes (redhat-lab-cube registers as RHLab-Houston, and staging-int as Staging-int — a case-only difference). A 2026-09-05 audit of all 172 environments confirmed every agent module carries a unique resourceId. Enrollment is decided by whether a Teleport kube agent is actually deployed, not by whether the agent module is linked — a cube can carry the module and only teleport-app-service, which is the expose pathway and grants no Kubernetes access. Such a cube reads Not enrolled rather than showing nothing — the group is labelled "Connectivity" and enumerates the systems consulted, so an absent chip would be indistinguishable from one still loading or failing. disconnected is grey for the same air-gap reason as the badge. See docs/console-spec.md §4.17. The cube page also carries a Resources tab (?tab=resources, so the view is bookmarkable) with four line graphs — CPU, Memory, Host disk and Volume fill — drawn from Groundcover range metrics and fetched only while that tab is open. No metric Groundcover exposes is cube-wide, so every figure is a server-side aggregation and is labelled as one: a series reads "Average across 5 hosts", never "Cube CPU utilization", and the entity count is shown beside each panel because an average silently improves when a host stops reporting. Volume fill leads with the fullest volume rather than the average — one volume at 99% among thirty barely moves a mean, so an averaged panel would render calm while an app is minutes from crashlooping on a full PV. There is no alarm colour and no polling: the busiest host legitimately sits near 100% on a working cube, and refresh is a button, because the single Groundcover key's rate budget is shared across the whole fleet. "We could not read it" and "this cube does not report it" are distinct panel states and never collapse into each other. See docs/console-spec.md §4.18. Opening a cube also shows its inventory identity — Customer, Cube label, Environment, Cube serial, Status and Location — read from the edgescale-inventory service, which Apollo does not carry, plus a Hardware tab with a specification table (server, motherboard, CPU, memory, storage, GPU) built from the same record at no extra request. Component serials, MAC addresses and firmware revisions are deliberately excluded from it — see docs/console-spec.md §4.13. It is its own endpoint fetched alongside the cube detail, and deliberately not joined onto the cube list: an inventory lookup is a ~3.3s server-side partition scan, so joining the list cost an admin ~45s on a previously-instant page — one lookup for the cube you open, instead of one per cube you can see. Gated separately by inventory_enabled (off by default) with two modes: api against the real service, or simulated, which serves clearly-badged fictional records so the feature works before the API credential exists. See docs/console-spec.md §4.13. Gated per-env by console_enabled (off by default; on in both dev and prod since the console launch) — that flag takes down the routes, the guide, and the apex cookie allowlist together, and the Cognito invite email now links to /auth/login, which needs it on. See docs/console-spec.md.

Install

Requires uv (or pipx).

# Add to Claude Code
claude mcp add cube -- npx @edgescaleai/cube-mcp

Or run directly:

npx @edgescaleai/cube-mcp    # via Node (calls uvx under the hood)
uvx edgescaleai-cube-mcp     # via uv directly

Getting Started

You: "Log me in"             → agent_login_browser (opens browser for Cognito login)
You: "Show Cube status"      → cube_status
You: "What pods are failing?" → kubectl_exec (runs kubectl server-side)

Tools

Auth

Kubernetes

Apps

App Exposure

Apps can be visible to several app companies (a partner and its customer, say), on a stack whose app_companies_source is stored; prod stays single-company until the transition. See docs/multi-company-app-access-spec.md.

The URL to hand out is the browser-app-proxy one these tools return — https://<app>-<cube>.<apex>. Every app is registered as a Teleport app under the hood, but Teleport's own …teleport.sh address requires a Teleport account, which we are retiring; it is stored in Apollo for Teleport's routing and is deliberately absent from every tool response and the console. A newly exposed URL starts answering ~80s after the call, once the cube rolls out the app-service release.

app_proxy is a separate, unrestricted capability: it port-forwards any exposed app to a local port for whatever wants one — psql, an MQTT client, scripting — and is the only way into a tcp:// app. It and the public URL are independent; both work at once. The expose response simply doesn't advertise it as the next step, because the published URL is what you hand someone.

app_uri must use http://, https:// or tcp://. Use tcp:// for anything that does not speak HTTP (MQTT, Postgres, an S3 API); such an app has no browser URL at all and is reached only through app_proxy. A wrong scheme used to be stored without complaint and surfaced much later as an unrelated-looking Teleport dial error, so it is now refused up front.

rewrite_headers covers services that need the forwarded request to look like something else — most often an nginx ingress that routes by Host, where the value must match the in-cluster ingress rule and is not derivable from the app's own address. It replaces the whole list rather than merging, so read the current values with list_exposed first; omitting the argument leaves an existing rewrite untouched, and [] removes it. Removal matters as much as setting: a rewrite that pins Host/Origin to a …teleport.sh address makes an app unreachable through its browser URL. Some apps additionally embed their own public URL in their own configuration (MinIO's CONSOLE_BROWSER_REDIRECT_URL, for one) — no rewrite header can fix that; those need an update_entity_config change as well.

Build & Registry

Remote (Docker-free) builds are driven server-side by three internal proxied tools — build_image_prepare, build_image_start, build_image_status — which build_and_publish_to_apollo orchestrates automatically. They are plumbing for the async submit→poll kaniko flow, not meant for direct invocation. See docs/remote-build-spec.md.

The pure-Python chart packager does not fetch chart dependencies — vendor any subcharts (e.g. helm dependency build) into charts/ before publishing, or Apollo rendering will fail on the missing dependency.

Apollo Environments

Apollo Modules

Apollo Secrets

Apollo Products & Release Channels

Groundcover (observability)

Read-only, per-cube observability. Access is scoped to the cubes your Apollo credentials can see, and every query is pinned server-side to a single cube — you cannot read another cube's data. See docs/groundcover-integration-spec.md.

Feedback

Raises a ticket on the EdgescaleAI Customer Feedback Jira board (project CF) so product feedback lands in the same queue the team already triages, without leaving the agent. Off unless the stack enables it (jira_enabled).

The Edgeport Console (/console) offers feedback through a "Send feedback" button, reachable from any page once signed in. It has two destinations: "Report a bug" files a Freshservice ticket (tools/freshservice.py), while feature requests raise a customer request on the Jira CF board. Note the console deliberately promises no reply or response time on either path, and does not offer to answer questions — the Freshservice integration is complete, but the support operations behind it (on-call, SLAs) are not, so the wording is held back until they are. See docs/console-spec.md §4.4e Deferred: support framing for the restore checklist. The split is deliberate: a helpdesk and a product backlog want different triage. The Freshservice tenant is edgescaleai.freshservice.com — note that is not edgeport.freshservice.com, which does not exist. Both paths resolve the reporter server-side and neither collects an email address — see docs/console-spec.md §4.4e. It calls straight into the same module rather than a second integration, and never collects an email address: the reporter is resolved from the signed-in session the same way it is resolved from the caller's API key on the MCP path.

Every ticket carries a label identifying which surface raised it, so agent-submitted, console-submitted, and portal/email traffic can be told apart:

project = CF AND labels = "submitted-via-mcp" ORDER BY created DESC
project = CF AND labels = "submitted-via-console" ORDER BY created DESC

The submitter is identified server-side from the calling API key, so feedback cannot be attributed to someone else by argument, and a normal browser-login session never puts an email address in the tool arguments. A bot or CLI key has no address on file and must pass contact_email, which is masked in telemetry.

Authentication is an Atlassian service account using an OAuth 2.0 client-credentials grant, holding exactly one scope — write:servicedesk-request. The credential can create requests and can read nothing at all; see Operations & Ownership for the secret and the ids it needs.

Neither surface returns a link to the created ticket — just its key (e.g. CF-123) as a reference number. The submitter is never given a JSM agent seat, so a Jira browse link would not actually open for them.

Local Development

# Install dependencies
uv sync --extra dev
uv pip install -e packages/cube-common -e packages/cube-cloud -e packages/cube-agent

# Run tests
uv run pytest packages/ -v

# Run cube-agent locally (for debugging)
uv run cube-agent

To test with Claude Code, point the MCP server at your local code:

claude mcp add cube-local -- uv run --directory /path/to/cube-mcp cube-agent

Reload after changes with /mcp in Claude Code.

claude mcp add's local scope is keyed per project path, so if you work across multiple git worktree checkouts of this repo, each one needs its own cube-local entry (or they collide on the name). .claude/cube-local.mcp.json avoids that by using --directory . instead of a hardcoded path, so the same file works unmodified from any checkout:

cd /path/to/cube-mcp  # or any worktree of it
claude --strict-mcp-config --mcp-config .claude/cube-local.mcp.json

--strict-mcp-config loads only the servers in that file, dropping any other MCP servers configured for the session — pass additional --mcp-config <file> arguments if you need those alongside it.

Contributing

cube-mcp uses a deploy-on-merge model: main is the release branch and deploys to prod; dev-main is a shared testing branch that deploys to the dev environment. New work is built on a branch cut from main, validated on the dev stack, and merged to main only once it works — feature PRs target main, not dev-main.

Branch and promotion flow

1. Branch from main. Cut your branch off the latest main and name it feature/<desc> or fix/<desc>:

git fetch origin
git switch -c feature/my-change origin/main

2. Open a PR into main. Make your changes, run uv run pytest packages/ -v, then open a PR targeting main (not dev-main). It should get a review and green CI before merging.

3. Validate on the dev stack first. Before merging, push your branch's changes to dev-main to deploy them to the dev environment (cube-mcp-dev-*) via deploy-dev.yml, and exercise the change end-to-end against real infra. dev-main is a testing ground — catch deploy- and infra-level problems here, not in prod. The feature PR still targets main.

4. Merge to main once verified. Only after the change works on dev, merge the main-targeted PR. Merging deploys to prod (deploy.yml) and, for publish-path changes, auto-publishes to PyPI + npm (publish.yml).

Promoting dev-main itself. Opening a dev-main → main promotion PR is acceptable only when dev-main holds exactly one unmerged feature; with more than one it bundles unverified changes into a single prod merge, so land those through their own main-targeted PRs instead. Because dev-main is a shared, disposable testing branch, it can be reset to main when it drifts or accumulates stale test commits — coordinate with anyone else testing on dev before doing so.

Operations & Ownership

This section exists so anyone inheriting cube-mcp can run it end-to-end without tribal knowledge. If something here is wrong or missing, fix it in this README — don't keep the truth in your head.

Runbooks (point Claude Code at these)

These docs are written in runbook style — open Claude Code in this repo and ask it to "follow docs/adding-new-user.md to add alice@example.com to the lear-dev group" and it will execute the AWS CLI calls itself.

Where AWS resources live

Everything is in AWS account 992382448282, region us-west-2. All resources are managed by Terraform in infra/terraform/ — change them there, not in the console.

Four telemetry flags gate the analytics stack — telemetry_enabled (S3 sink), telemetry_compaction_enabled (nightly Lambda), otlp_enabled (Groundcover spans), and telemetry_console_enabled (console usage events through the same sinks, under source=console; a sub-switch beneath telemetry_enabled). All default off and are set per-env in infra/terraform/environments/*.tfvars; all four are true in both dev and prod.

Outside AWS:

• Teleport: edgescaleai.teleport.sh — bot is cube-mcp-bot, joins via IAM. See docs/admin.md §4.
• Apollo: edgescaleai.palantirapollo.com — OAuth2 credentials per profile, stored in Secrets Manager.
• Groundcover: api.groundcover.com — one workspace/tenant for the whole fleet (each cube is a cluster); we enforce per-cube tenancy ourselves. Key in Secrets Manager (cube-mcp/groundcover/api-key).
• Jira Service Management: edgescaleai.atlassian.net (cloud id e9f7f215-8e22-4d9e-b4d0-12e11739b67d, service desk 100 = project CF). Service-account OAuth 2.0 client credentials in Secrets Manager (cube-mcp/jira/oauth), scope write:servicedesk-request. The credential cannot read Jira, so the request-type and field ids in tools/feedback.py are configuration — if the CF forms change, they go stale and submissions 400.
• Cloudflare (DNS): the public domains (cube.edgescaleai-cube.com / cube-dev.edgescaleai-cube.com) are DNS-only CNAMEs → the ALB, managed manually in Cloudflare, not Terraform (the aws_route53_records are no-ops). The browser app proxy additionally needs a wildcard *.cube.edgescaleai-cube.com DNS-only CNAME → the prod ALB; until it's added the feature ships dormant (subdomains won't resolve). See docs/browser-app-proxy-spec.md Appendix B.
• PyPI / npm: auto-published from main via CI (GitHub Actions). The npm shim under npm/ is a thin wrapper that shells to uvx edgescaleai-cube-mcp.

CI/CD pipelines

All deploys go through GitHub Actions in .github/workflows/. AWS auth uses OIDC — no long-lived AWS keys are stored in GitHub.

OIDC role for all AWS-touching jobs: arn:aws:iam::992382448282:role/github-actions.

Required CI secrets

Stored as GitHub repo secrets. To rotate any of these, generate a new value at the source and update via gh secret set <NAME> (or the repo Settings → Secrets UI).

Rotate the publish tokens BEFORE merging anything that triggers publish.yml. That workflow bumps the version, commits Release vX.Y.Z, and pushes the tag to main before it uploads — so an invalid token leaves a tagged release commit on main with nothing on PyPI or npm. Verify first (see "Verifying a publish token" below).

Verifying a publish token

deploy-dev.yml publishes to the same PyPI project and npm package as a release, just with a timestamped .devN / -dev.N version — so dev-main is the safe way to exercise both tokens without cutting a release. Note the workflow has a paths: filter, so an empty commit will not trigger it; dispatch it instead:

gh workflow run "Deploy Dev" --repo EdgescaleAI/cube-mcp --ref dev-main

Publish dev packages (PyPI + npm) going green means both tokens work. 403 Forbidden from twine means the PyPI token is wrong-scoped or the account lacks ownership; E401 / ENEEDAUTH from npm means the same on that side. Be aware the dispatch runs the whole workflow, so it also re-applies Terraform and redeploys dev. To check credentials alone, without publishing:

cd npm && NODE_AUTH_TOKEN='npm_…' npm publish --dry-run --access public   # authenticates, no version created
TWINE_USERNAME=__token__ TWINE_PASSWORD='pypi-…' twine upload --repository testpypi dist/*

Confirm the artifact actually landed, and — on npm — who published it, which is how you prove CI is no longer using a departed maintainer's credential:

curl -s https://registry.npmjs.org/@edgescaleai%2Fcube-mcp \
  | python3 -c "import json,sys; d=json.load(sys.stdin); v=d['dist-tags']['dev']; \
print(v, d['time'][v], (d['versions'][v].get('_npmUser') or {}).get('name'))"

Offboarding a maintainer

Rotating PYPI_TOKEN / NPM_TOKEN changes only what CI uses. It does not revoke the departing person's own access — their personal tokens keep publish rights until they are removed from the package itself. Both steps are required:

npm owner rm <their-npm-user> @edgescaleai/cube-mcp

and remove them on the PyPI collaboration page. Also revoke their old tokens at the source while you still have access to their accounts.

Worth doing instead of rotating again: both registries support OIDC trusted publishing, where GitHub Actions mints a short-lived credential per run. That removes PYPI_TOKEN and NPM_TOKEN entirely — nothing to rotate, and nothing to inherit when someone leaves. It needs a permissions: id-token: write block on the publishing jobs plus a publisher configured on each registry.

AWS credentials are not stored as secrets — deploy.yml/deploy-dev.yml assume arn:aws:iam::992382448282:role/github-actions via OIDC. To grant a new repo or change permissions, update that role's trust policy and inline policies in AWS IAM.

Terraform state

Stored in S3, no DynamoDB lock table currently configured (single-writer assumption — CI is the only applier).

Local apply (only if you really need to bypass CI):

cd infra/terraform
terraform init                                                    # prod
terraform init -backend-config="key=cube-cloud-dev/terraform.tfstate"  # dev
terraform plan

The S3 bucket has versioning enabled, so a corrupt state file can be rolled back to a previous version via the AWS console.

Dev environment

A parallel stack exists in the same AWS account for testing infra changes before merging to main.

dev-main is a testing environment first: push your branch's changes there to deploy them to the dev stack and validate before they land on main. Feature PRs target main directly and are merged only after the change has been verified on the dev stack. Promoting dev-main → main as a single PR is acceptable only when dev-main holds exactly one unmerged feature; with more than one it would bundle unverified changes into a single prod merge, so promote those individually via their own main-targeted PRs. See Contributing for the full branch-and-promotion flow.

Versioning & releases

• Scheme: patch-bump only, automated. publish.yml reads pyproject.toml, increments the patch, writes it back to both pyproject.toml and npm/package.json, commits as github-actions[bot], and tags vX.Y.Z.
• Major/minor bumps: edit pyproject.toml manually in a PR. The next merge to main that touches src/ / npm/ / pyproject.toml will publish from there.
• PyPI ↔ npm sync: the same version number is used for both packages. The npm/ shim is a thin wrapper that shells to uvx edgescaleai-cube-mcp.
• What "main" means for users: every merge that touches the publish paths ships to PyPI + npm within minutes. There is no staging release.

Rollback playbook

Monitoring & alerting

Alerting: not yet configured. No CloudWatch alarms, SNS topics, Sentry, or paging are set up. Nothing pages anyone when the service degrades — that is still the biggest gap.

Observability that does exist: CloudWatch Logs (/ecs/cube-mcp-prod) + ECS task health for the service itself; tool-call analytics telemetry (live in prod) recording every cloud tool call — and, where telemetry_console_enabled is on, every counted console request as a source=console row of the same shape — to two sinks — S3 Parquet (durable analytics, queryable with the DuckDB query pack in docs/tool-call-analytics-queries.md) and Groundcover OTLP spans (hot/correlation tier). See docs/tool-call-logging-spec.md. Per-cube observability (of the fleet, not this service) is available via the groundcover_* tools.

Day-to-day debugging:

# Live tail prod logs (cube-cloud + tbot)
aws logs tail /ecs/cube-mcp-prod --follow --region us-west-2

# Filter to just tbot
aws logs tail /ecs/cube-mcp-prod --filter-pattern "tbot" --follow

# Service health
aws ecs describe-services \
  --cluster cube-mcp-prod-cluster \
  --services cube-mcp-prod-service \
  --query 'services[0].{running:runningCount,desired:desiredCount,events:events[0:5]}'

Recommended additions for the next owner (none of these exist yet):

• CloudWatch alarm on ECS service RunningTaskCount < 1 → SNS → email/PagerDuty.
• ALB 5xx alarm on the listener.
• DynamoDB throttling alarm on cube-mcp-prod-api-keys.
• Synthetic check that runs agent_login_browser end-to-end weekly.

Stakeholders & contacts

TODO(owner): Vinayak (departing) is the only person with full context on these. Fill these in before he leaves so the next owner has someone to call.

Cost & billing

TODO(owner): Add rough monthly run-rate and the account that gets billed.

• AWS (account 992382448282): TODO — typical monthly spend, biggest line items, who pays the invoice.
• Teleport (edgescaleai.teleport.sh): TODO — plan tier, seat count, billing contact.
• Apollo (edgescaleai.palantirapollo.com): TODO — Palantir contract reference, who renews.
• Domain (edgescaleai-cube.com): TODO — registrar, expiry date, who has the login.
• PyPI / npm: free tiers, no recurring cost.

If costs need to be cut quickly: the dev ECS service (cube-mcp-dev-service) can be scaled to zero with no user impact.

Access & disaster recovery

Known tech debt / planned work

Update this list as you ship or de-scope items so the next owner sees current state, not folklore.

• Phase 2 RBAC: per-profile Apollo RBAC is now implemented and validated — each scoped profile has its own Apollo SA + team + creds in Secrets Manager (validated end-to-end through MCP: lear-sandbox, pltr-perception-dev, operations). Remaining work: set up and validate lear-dev / conagra-dev / pltr-dev / edgescaleai-dev (real Teleport roles) through the MCP pipeline. Provisioning is done by scripts/provision-access.py.
• ECR :latest-only tagging makes ECS rollback awkward (see Rollback playbook). Tag images with the git SHA in deploy.yml.
• No alerting. See Monitoring section.
• Teleport role management is manual tctl. The future plan is the Teleport Terraform provider — see docs/admin.md → Future.
• No DynamoDB lock on Terraform state. Acceptable while CI is the only applier; add a lock table if local applies become routine.
• Cognito drift: esaiadmin group. Surfaced in #93. The prod Cognito pool has an esaiadmin group that isn't defined in infra/terraform/cognito.tf — it was created in the console. Either terraform import it or delete it; don't leave it as drift.
• TODO(owner): anything else in flight at handoff time.

First-day checklist for a new owner

1. Read docs/admin.md end-to-end. Then run agent_login_browser yourself so you've experienced the user flow.
2. Get console access to AWS account 992382448282 (us-west-2) with permissions for Cognito, DynamoDB, Secrets Manager, ECS, IAM, and CloudWatch Logs.
3. Get Teleport admin (tctl) on edgescaleai.teleport.sh — needed to manage the cube-mcp-bot and add roles. Ask the person listed in Stakeholders & contacts.
4. Get Apollo admin on edgescaleai.palantirapollo.com — needed to issue per-profile OAuth2 credentials when adding new tenants.
5. Install cube-admin locally for API key management: uv pip install -e packages/cube-admin. This package is internal-only and must never be published.
6. Verify CI is healthy and you can rotate every secret in Required CI secrets. Push a no-op commit to dev-main to confirm deploy-dev.yml succeeds end-to-end against the dev stack.
7. Replace every TODO(owner) in this README — Stakeholders, Cost, Access & DR, Tech debt. If you can't fill one in, write down who you asked and what they said. Don't leave them blank.

Common ops tasks

• Provision access (recommended): uv run scripts/provision-access.py profile create <name> --cubes <id1,id2> then uv run scripts/provision-access.py member add <email> <name> — the access CLI (Apollo SA/team/roles + Teleport role/bot + tbot identity in Secrets Manager + Cognito group/members), zero-setup, dry-run by default with a per-facet plan; add --apply to execute. Scope changes: profile add-cubes / remove-cubes / set-cubes. App companies (which exposed apps a profile sees): profile set-app-companies / add-app-companies / remove-app-companies — no code change or deploy.
• Look up access: uv run scripts/provision-access.py member show <email> (a user's profile and reach) and uv run scripts/provision-access.py profile show <name> / profile list (every facet of a profile, drift flagged).
• Add a user: docs/adding-new-user.md.
• Add a new tenant / role: docs/admin.md → Creating a new role / profile. Prefer scripts/provision-access.py profile create; the six-step manual flow (Cognito group, SM secret, Teleport role, bot role grant, tbot output, merge) is kept as break-glass.
• Revoke access: docs/admin.md → Revoking access.
• Manage API keys: cube-admin keys {create,list,revoke} — see docs/admin.md → Managing API keys with cube-admin.
• Debug a failing tool call: check aws logs tail /ecs/cube-mcp-prod --follow for the cube-cloud and tbot containers.

Repo layout cheatsheet

packages/
  cube-agent/    # Local MCP server (npx @edgescaleai/cube-mcp entrypoint)
  cube-cloud/    # FastAPI service on ECS — auth, RBAC, Apollo, Teleport relay
  cube-common/   # Shared types and helpers
  cube-admin/    # Internal-only API key CLI — DO NOT PUBLISH
infra/
  terraform/     # All AWS resources
  tbot/          # tbot Dockerfile, entrypoint, prod config
docs/            # Runbooks (start here)
resources/       # User-facing reference docs
npm/             # npm shim that calls uvx edgescaleai-cube-mcp
scripts/         # Release / publish helpers