cube-mcp

MCP server for EdgescaleAI Cube management and Apollo deployments.

New owner / inheriting this repo? Jump to Operations & Ownership. The two runbooks you need are docs/admin.md (full RBAC + infra reference) and docs/adding-new-user.md (onboard a teammate end-to-end). Both are written so you can point Claude Code at them and have it execute the steps for you.

Architecture

cube-agent runs locally as an MCP server (via npx @edgescaleai/cube-mcp). It handles authentication, Docker builds, Helm chart packaging, and app proxy tunnels. Server-side operations (Kubernetes, Teleport, Apollo, Groundcover observability) are proxied to cube-cloud, a FastAPI backend hosted on AWS ECS.

┌──────────────────────────────────────────────────────────────────────────────┐
│  LOCAL (developer machine)                                                   │
│                                                                              │
│  ┌─────────────┐    MCP JSON-RPC     ┌──────────────────────────────────┐   │
│  │ Claude Code  │◄──────────────────►│  cube-agent (MCP server)         │   │
│  └─────────────┘                     │                                  │   │
│                                      │  Local tools:                    │   │
│                                      │   • agent_login_browser          │   │
│                                      │   • agent_logout                 │   │
│                                      │   • agent_status                 │   │
│                                      │   • build_and_publish_to_apollo  │   │
│                                      │   • app_proxy / stop / status    │   │
│                                      │                                  │   │
│                                      │  Remote tools:                   │   │
│                                      │   (proxied to cube-cloud ──────) │   │
│                                      └────────┬──────────┬──────────────┘   │
│                                               │          │                   │
│   ~/.cube-agent/  ◄── API key storage         │          │ WebSocket         │
│   localhost:PORT  ◄── app proxy listener      │          │ /tunnel           │
│                                               │          │                   │
└───────────────────────────────────────────────┼──────────┼───────────────────┘
                                                │          │
                               HTTPS + Bearer   │          │  TCP-over-WS
                               POST /mcp/        │          │  relay
                                                │          │
┌───────────────────────────────────────────────┼──────────┼───────────────────┐
│  CLOUD (AWS ECS)                              │          │                   │
│                                               ▼          ▼                   │
│  ┌────────────────────────────────────────────────────────────────────────┐  │
│  │  cube-cloud (FastAPI)                                                  │  │
│  │                                                                        │  │
│  │  ┌─────────────────────┐     ┌──────────────────────────────────────┐ │  │
│  │  │  Auth Middleware     │     │  RBAC (Cognito + Profiles)          │ │  │
│  │  │  • Validate API key  │────►│  • Cognito group → profile          │ │  │
│  │  │    (DynamoDB lookup)│     │  • Profile → Apollo credentials     │ │  │
│  │  │  • Attach profile   │     │    (AWS Secrets Manager)            │ │  │
│  │  └─────────────────────┘     └──────────────────────────────────────┘ │  │
│  │                                                                        │  │
│  │  Cloud tools (RBAC-gated):                                            │  │
│  │  Kubernetes        Apollo Environments      Apollo Products           │  │
│  │  • cube_list       • list_environments      • list_products           │  │
│  │  • cube_status     • install_entity         • compare_product_        │  │
│  │  • kubectl_exec    • uninstall_entity         versions                │  │
│  │  • app_list        • entity_health          • list_release_channels   │  │
│  │  • list_services   • plan_details           • get_product_releases    │  │
│  │                    • cancel_active_plan                               │  │
│  │  Expose (Teleport) • update_entity_config   Groundcover (read-only)   │  │
│  │  • expose          • enforce_entity_config  • groundcover_list_*      │  │
│  │  • unexpose                                 • groundcover_query_*     │  │
│  │  • list_exposed    Modules                  • groundcover_search_*    │  │
│  │                    • list_modules           • groundcover_k8s_events  │  │
│  │  Secrets           • install_module                                   │  │
│  │  • create_secret   • uninstall_module       Registry                  │  │
│  │  • update_secret   • update_module_         • acr_get_token           │  │
│  │                      variables                                        │  │
│  └────────────────────────┬───────────────────────────┬──────────────────┘  │
│                           │                           │                      │
│  ┌────────────────────────┴────────────────┐          │                      │
│  │  tbot sidecar (Teleport credentials)    │          │                      │
│  │  • IAM join method (Fargate identity)   │          │                      │
│  │  • identity/  → tsh commands            │          │                      │
│  │  • kube/{cluster}/ → kubeconfigs        │          │                      │
│  │  • app-{name}/ → app TLS certs          │          │                      │
│  └──────────┬─────────────────────────────┘          │                      │
│             │                                         │                      │
└─────────────┼─────────────────────────────────────────┼──────────────────────┘
              │                                         │
              │ tsh/tctl (short-lived certs)            │ GraphQL (OAuth2)
              ▼                                         ▼
┌──────────────────────────┐            ┌──────────────────────────────────┐
│  Teleport                │            │  Apollo                          │
│  edgescaleai.teleport.sh │            │  edgescaleai.palantirapollo.com  │
│                          │            │                                  │
│  • Kube clusters         │            │  • Environments & Modules        │
│  • App proxies           │            │  • Entities (Helm charts)        │
│  • SSH access            │            │  • Products & Release Channels   │
│  • Identity & TLS certs  │            │  • Change Requests               │
└──────────────────────────┘            │  • ACR (Docker + Helm registry)  │
                                        └──────────────────────────────────┘

                        │ HTTPS (per-cube, read-only)
                        ▼
            ┌──────────────────────────────────┐
            │  Groundcover                     │
            │  api.groundcover.com             │
            │  • Metrics / logs / traces       │
            │  • K8s events & workload health  │
            └──────────────────────────────────┘

A separate browser front door (not shown above) serves `expose`d in-cube web
apps to account-less external users: browser → `<app>-<env>` wildcard subdomain
→ ALB → cube-cloud browser proxy → tbot mTLS dial into the cube. See
[docs/browser-app-proxy-spec.md](docs/browser-app-proxy-spec.md).

How it works

1. Local tools run directly on the developer's machine inside cube-agent. These handle authentication (Cognito browser login, API key storage), Docker builds, Helm chart packaging, and app proxy tunnels.

2. Cloud tools are proxied from cube-agent to cube-cloud (FastAPI on AWS ECS) via MCP Streamable HTTP. The MCP endpoint (/mcp) and the tunnel relay (/tunnel) are authenticated with a Bearer API key validated against DynamoDB. (The browser surfaces additionally accept the same key from a parent-domain session cookie — on app subdomains, and on the apex under the /console prefix only; /mcp and /tunnel stay Bearer-only as a CSRF gate.)

3. RBAC maps Cognito user groups to profiles. Each profile resolves to Apollo OAuth2 credentials stored in AWS Secrets Manager, scoping what the user can access.

4. Teleport access is provided by a tbot sidecar running alongside cube-cloud in the same ECS task. It uses IAM join to obtain short-lived certificates for Kubernetes clusters and app proxies — no tsh is needed on the developer's machine.

5. Apollo operations (environments, modules, entities, products, releases, change requests) go through a GraphQL API authenticated with per-profile OAuth2 client credentials.

6. Groundcover provides read-only, per-cube observability (metrics, logs, traces, K8s events, workload health) via the groundcover_* tools. Access is scoped to the cubes your Apollo credentials can see, and every query is pinned server-side to a single cube — see docs/groundcover-integration-spec.md.

7. Browser app proxy gives account-less external users a browser-only path to in-cube web apps you've exposed — no Teleport account and no local agent. Requests hit an <app>-<env> wildcard subdomain, authenticate via a Cognito session cookie, and are dialed into the cube over the same tbot mTLS path. See docs/browser-app-proxy-spec.md.

8. Console is the browser UI at https://<apex>/console — sign in at the apex and you land on it. A React SPA (built from console/web, compiled by a node stage in the cube-cloud Dockerfile) over JSON endpoints that share the Apollo query + RBAC layer with the MCP tools. Cookie-authenticated, bookmarkable, and deliberately narrow: reads are GET-only and same-origin-enforced, and there is no generic tool-proxy endpoint. Each app card links the app's public browser-app-proxy URL (https://<app>-<cube>.<apex> — not Teleport's …teleport.sh address, which needs a Teleport account), and offers an inline Unexpose behind a confirmation that names the cube. Each app also links a Logs view at its own bookmarkable URL (/console/cubes/<cube>/apps/<app>/logs), backed by Groundcover — one query per Kubernetes workload the app owns, merged newest-first, and near-live: it polls every 10s incrementally (only rows newer than the last window), pauses when the tab is hidden, and offers a Live/Paused toggle. Streaming was deliberately not used — Groundcover exposes only search endpoints, so a WebSocket would still poll underneath while adding a second auth path. Each app row also offers Expose, which lists the cube's Kubernetes Services as candidates (auto-selecting the port when a Service has exactly one, and requiring a choice otherwise — a wrong port surfaces later as a Teleport timeout) and refuses any target the console did not itself enumerate. Expose and unexpose are the console's only mutations: it is its own route with enumerated args, gated on origin + a X-Console-CSRF synchronizer token, and it fails closed where the CSRF secret is unprovisioned — which is the case in prod today. Gated per-env by console_enabled (dev only today; off in prod, and off by default) — that flag takes down both the routes and the apex cookie allowlist together. See docs/console-spec.md.

Install

Requires uv (or pipx).

# Add to Claude Code
claude mcp add cube -- npx @edgescaleai/cube-mcp

Or run directly:

npx @edgescaleai/cube-mcp    # via Node (calls uvx under the hood)
uvx edgescaleai-cube-mcp     # via uv directly

Getting Started

You: "Log me in"             → agent_login_browser (opens browser for Cognito login)
You: "Show Cube status"      → cube_status
You: "What pods are failing?" → kubectl_exec (runs kubectl server-side)

Tools

Auth

Kubernetes

Apps

App Exposure (Teleport)

Build & Registry

Remote (Docker-free) builds are driven server-side by three internal proxied tools — build_image_prepare, build_image_start, build_image_status — which build_and_publish_to_apollo orchestrates automatically. They are plumbing for the async submit→poll kaniko flow, not meant for direct invocation. See docs/remote-build-spec.md.

The pure-Python chart packager does not fetch chart dependencies — vendor any subcharts (e.g. helm dependency build) into charts/ before publishing, or Apollo rendering will fail on the missing dependency.

Apollo Environments

Apollo Modules

Apollo Secrets

Apollo Products & Release Channels

Groundcover (observability)

Read-only, per-cube observability. Access is scoped to the cubes your Apollo credentials can see, and every query is pinned server-side to a single cube — you cannot read another cube's data. See docs/groundcover-integration-spec.md.

Local Development

# Install dependencies
uv sync --extra dev
uv pip install -e packages/cube-common -e packages/cube-cloud -e packages/cube-agent

# Run tests
uv run pytest packages/ -v

# Run cube-agent locally (for debugging)
uv run cube-agent

To test with Claude Code, point the MCP server at your local code:

claude mcp add cube-local -- uv run --directory /path/to/cube-mcp cube-agent

Reload after changes with /mcp in Claude Code.

claude mcp add's local scope is keyed per project path, so if you work across multiple git worktree checkouts of this repo, each one needs its own cube-local entry (or they collide on the name). .claude/cube-local.mcp.json avoids that by using --directory . instead of a hardcoded path, so the same file works unmodified from any checkout:

cd /path/to/cube-mcp  # or any worktree of it
claude --strict-mcp-config --mcp-config .claude/cube-local.mcp.json

--strict-mcp-config loads only the servers in that file, dropping any other MCP servers configured for the session — pass additional --mcp-config <file> arguments if you need those alongside it.

Contributing

cube-mcp uses a deploy-on-merge model: main is the release branch and deploys to prod; dev-main is a shared testing branch that deploys to the dev environment. New work is built on a branch cut from main, validated on the dev stack, and merged to main only once it works — feature PRs target main, not dev-main.

Branch and promotion flow

1. Branch from main. Cut your branch off the latest main and name it feature/<desc> or fix/<desc>:

git fetch origin
git switch -c feature/my-change origin/main

2. Open a PR into main. Make your changes, run uv run pytest packages/ -v, then open a PR targeting main (not dev-main). It should get a review and green CI before merging.

3. Validate on the dev stack first. Before merging, push your branch's changes to dev-main to deploy them to the dev environment (cube-mcp-dev-*) via deploy-dev.yml, and exercise the change end-to-end against real infra. dev-main is a testing ground — catch deploy- and infra-level problems here, not in prod. The feature PR still targets main.

4. Merge to main once verified. Only after the change works on dev, merge the main-targeted PR. Merging deploys to prod (deploy.yml) and, for publish-path changes, auto-publishes to PyPI + npm (publish.yml).

Promoting dev-main itself. Opening a dev-main → main promotion PR is acceptable only when dev-main holds exactly one unmerged feature; with more than one it bundles unverified changes into a single prod merge, so land those through their own main-targeted PRs instead. Because dev-main is a shared, disposable testing branch, it can be reset to main when it drifts or accumulates stale test commits — coordinate with anyone else testing on dev before doing so.

Operations & Ownership

This section exists so anyone inheriting cube-mcp can run it end-to-end without tribal knowledge. If something here is wrong or missing, fix it in this README — don't keep the truth in your head.

Runbooks (point Claude Code at these)

These docs are written in runbook style — open Claude Code in this repo and ask it to "follow docs/adding-new-user.md to add alice@example.com to the lear-dev group" and it will execute the AWS CLI calls itself.

Where AWS resources live

Everything is in AWS account 992382448282, region us-west-2. All resources are managed by Terraform in infra/terraform/ — change them there, not in the console.

Three telemetry flags gate the tool-call analytics stack — telemetry_enabled (S3 sink), telemetry_compaction_enabled (nightly Lambda), and otlp_enabled (Groundcover spans). All default off and are set per-env in infra/terraform/environments/*.tfvars; all three are true in prod.

Outside AWS:

• Teleport: edgescaleai.teleport.sh — bot is cube-mcp-bot, joins via IAM. See docs/admin.md §4.
• Apollo: edgescaleai.palantirapollo.com — OAuth2 credentials per profile, stored in Secrets Manager.
• Groundcover: api.groundcover.com — one workspace/tenant for the whole fleet (each cube is a cluster); we enforce per-cube tenancy ourselves. Key in Secrets Manager (cube-mcp/groundcover/api-key).
• Cloudflare (DNS): the public domains (cube.edgescaleai-cube.com / cube-dev.edgescaleai-cube.com) are DNS-only CNAMEs → the ALB, managed manually in Cloudflare, not Terraform (the aws_route53_records are no-ops). The browser app proxy additionally needs a wildcard *.cube.edgescaleai-cube.com DNS-only CNAME → the prod ALB; until it's added the feature ships dormant (subdomains won't resolve). See docs/browser-app-proxy-spec.md Appendix B.
• PyPI / npm: auto-published from main via CI (GitHub Actions). The npm shim under npm/ is a thin wrapper that shells to uvx edgescaleai-cube-mcp.

CI/CD pipelines

All deploys go through GitHub Actions in .github/workflows/. AWS auth uses OIDC — no long-lived AWS keys are stored in GitHub.

OIDC role for all AWS-touching jobs: arn:aws:iam::992382448282:role/github-actions.

Required CI secrets

Stored as GitHub repo secrets. To rotate any of these, generate a new value at the source and update via gh secret set <NAME> (or the repo Settings → Secrets UI).

AWS credentials are not stored as secrets — deploy.yml/deploy-dev.yml assume arn:aws:iam::992382448282:role/github-actions via OIDC. To grant a new repo or change permissions, update that role's trust policy and inline policies in AWS IAM.

Terraform state

Stored in S3, no DynamoDB lock table currently configured (single-writer assumption — CI is the only applier).

Local apply (only if you really need to bypass CI):

cd infra/terraform
terraform init                                                    # prod
terraform init -backend-config="key=cube-cloud-dev/terraform.tfstate"  # dev
terraform plan

The S3 bucket has versioning enabled, so a corrupt state file can be rolled back to a previous version via the AWS console.

Dev environment

A parallel stack exists in the same AWS account for testing infra changes before merging to main.

dev-main is a testing environment first: push your branch's changes there to deploy them to the dev stack and validate before they land on main. Feature PRs target main directly and are merged only after the change has been verified on the dev stack. Promoting dev-main → main as a single PR is acceptable only when dev-main holds exactly one unmerged feature; with more than one it would bundle unverified changes into a single prod merge, so promote those individually via their own main-targeted PRs. See Contributing for the full branch-and-promotion flow.

Versioning & releases

• Scheme: patch-bump only, automated. publish.yml reads pyproject.toml, increments the patch, writes it back to both pyproject.toml and npm/package.json, commits as github-actions[bot], and tags vX.Y.Z.
• Major/minor bumps: edit pyproject.toml manually in a PR. The next merge to main that touches src/ / npm/ / pyproject.toml will publish from there.
• PyPI ↔ npm sync: the same version number is used for both packages. The npm/ shim is a thin wrapper that shells to uvx edgescaleai-cube-mcp.
• What "main" means for users: every merge that touches the publish paths ships to PyPI + npm within minutes. There is no staging release.

Rollback playbook

Monitoring & alerting

Alerting: not yet configured. No CloudWatch alarms, SNS topics, Sentry, or paging are set up. Nothing pages anyone when the service degrades — that is still the biggest gap.

Observability that does exist: CloudWatch Logs (/ecs/cube-mcp-prod) + ECS task health for the service itself; tool-call analytics telemetry (live in prod) recording every cloud tool call to two sinks — S3 Parquet (durable analytics, queryable with the DuckDB query pack in docs/tool-call-analytics-queries.md) and Groundcover OTLP spans (hot/correlation tier). See docs/tool-call-logging-spec.md. Per-cube observability (of the fleet, not this service) is available via the groundcover_* tools.

Day-to-day debugging:

# Live tail prod logs (cube-cloud + tbot)
aws logs tail /ecs/cube-mcp-prod --follow --region us-west-2

# Filter to just tbot
aws logs tail /ecs/cube-mcp-prod --filter-pattern "tbot" --follow

# Service health
aws ecs describe-services \
  --cluster cube-mcp-prod-cluster \
  --services cube-mcp-prod-service \
  --query 'services[0].{running:runningCount,desired:desiredCount,events:events[0:5]}'

Recommended additions for the next owner (none of these exist yet):

• CloudWatch alarm on ECS service RunningTaskCount < 1 → SNS → email/PagerDuty.
• ALB 5xx alarm on the listener.
• DynamoDB throttling alarm on cube-mcp-prod-api-keys.
• Synthetic check that runs agent_login_browser end-to-end weekly.

Stakeholders & contacts

TODO(owner): Vinayak (departing) is the only person with full context on these. Fill these in before he leaves so the next owner has someone to call.

Cost & billing

TODO(owner): Add rough monthly run-rate and the account that gets billed.

• AWS (account 992382448282): TODO — typical monthly spend, biggest line items, who pays the invoice.
• Teleport (edgescaleai.teleport.sh): TODO — plan tier, seat count, billing contact.
• Apollo (edgescaleai.palantirapollo.com): TODO — Palantir contract reference, who renews.
• Domain (edgescaleai-cube.com): TODO — registrar, expiry date, who has the login.
• PyPI / npm: free tiers, no recurring cost.

If costs need to be cut quickly: the dev ECS service (cube-mcp-dev-service) can be scaled to zero with no user impact.

Access & disaster recovery

Known tech debt / planned work

Update this list as you ship or de-scope items so the next owner sees current state, not folklore.

• Phase 2 RBAC: per-profile Apollo RBAC is now implemented and validated — each scoped profile has its own Apollo SA + team + creds in Secrets Manager (validated end-to-end through MCP: lear-sandbox, pltr-perception-dev, operations). Remaining work: set up and validate lear-dev / conagra-dev / pltr-dev / edgescaleai-dev (real Teleport roles) through the MCP pipeline. Provisioning is done by scripts/provision-access.py.
• ECR :latest-only tagging makes ECS rollback awkward (see Rollback playbook). Tag images with the git SHA in deploy.yml.
• No alerting. See Monitoring section.
• Teleport role management is manual tctl. The future plan is the Teleport Terraform provider — see docs/admin.md → Future.
• No DynamoDB lock on Terraform state. Acceptable while CI is the only applier; add a lock table if local applies become routine.
• Cognito drift: esaiadmin group. Surfaced in #93. The prod Cognito pool has an esaiadmin group that isn't defined in infra/terraform/cognito.tf — it was created in the console. Either terraform import it or delete it; don't leave it as drift.
• TODO(owner): anything else in flight at handoff time.

First-day checklist for a new owner

1. Read docs/admin.md end-to-end. Then run agent_login_browser yourself so you've experienced the user flow.
2. Get console access to AWS account 992382448282 (us-west-2) with permissions for Cognito, DynamoDB, Secrets Manager, ECS, IAM, and CloudWatch Logs.
3. Get Teleport admin (tctl) on edgescaleai.teleport.sh — needed to manage the cube-mcp-bot and add roles. Ask the person listed in Stakeholders & contacts.
4. Get Apollo admin on edgescaleai.palantirapollo.com — needed to issue per-profile OAuth2 credentials when adding new tenants.
5. Install cube-admin locally for API key management: uv pip install -e packages/cube-admin. This package is internal-only and must never be published.
6. Verify CI is healthy and you can rotate every secret in Required CI secrets. Push a no-op commit to dev-main to confirm deploy-dev.yml succeeds end-to-end against the dev stack.
7. Replace every TODO(owner) in this README — Stakeholders, Cost, Access & DR, Tech debt. If you can't fill one in, write down who you asked and what they said. Don't leave them blank.

Common ops tasks

• Provision access (recommended): uv run python scripts/provision-access.py --email <email> --profile <name> --cubes <id1,id2> [--env prod|dev] [--apply] — full end-to-end provisioner (Apollo SA/team/roles + Teleport role/bot + tbot identity in Secrets Manager + Cognito group), dry-run by default, idempotent.
• Look up access: uv run python scripts/user-access.py show <email> [--env prod|dev] — prints the user's groups, resolved profile, and Apollo-accessible cubes (groups / grant subcommands too).
• Add a user: docs/adding-new-user.md.
• Add a new tenant / role: docs/admin.md → Creating a new role / profile. Prefer scripts/provision-access.py; the six-step manual flow (Cognito group, SM secret, Teleport role, bot role grant, tbot output, merge) is kept as break-glass.
• Revoke access: docs/admin.md → Revoking access.
• Manage API keys: cube-admin keys {create,list,revoke} — see docs/admin.md → Managing API keys with cube-admin.
• Debug a failing tool call: check aws logs tail /ecs/cube-mcp-prod --follow for the cube-cloud and tbot containers.

Repo layout cheatsheet

packages/
  cube-agent/    # Local MCP server (npx @edgescaleai/cube-mcp entrypoint)
  cube-cloud/    # FastAPI service on ECS — auth, RBAC, Apollo, Teleport relay
  cube-common/   # Shared types and helpers
  cube-admin/    # Internal-only API key CLI — DO NOT PUBLISH
infra/
  terraform/     # All AWS resources
  tbot/          # tbot Dockerfile, entrypoint, prod config
docs/            # Runbooks (start here)
resources/       # User-facing reference docs
npm/             # npm shim that calls uvx edgescaleai-cube-mcp
scripts/         # Release / publish helpers