tai42-e2e

The cross-repo, real-stack, end-to-end functional test suite for the tai / AgenticOS ecosystem. It boots the REAL deployment topology — a multi-worker uvicorn skeleton server, a separate backend worker process, a separate metrics server process, real Redis, real Postgres — and drives it over HTTP.

Why this suite exists. Every member suite (tai42-contract, tai42-kit, tai42-toolbox, tai42-agents, tai42-skeleton) is single-process, in-memory, and FakeRedis-backed by design, and stays that way. That leaves a whole class of bug invisible: the ones that live in the seam between processes — env-var ordering across a spawn boundary, a registry mutated on one worker and read on another, a socket flipped process-wide, a counter written in one process and scraped from another. tai-e2e is the only suite that can catch them, because it is the only one that runs the real multi-process topology.

Quickstart

docker compose up -d          # core profile: redis:7-alpine + postgres:16
uv python install 3.13
uv sync
uv run --no-sync pytest       # the default variant leg: arq + redis identity + local storage

The suite runs sequentially in one pytest process (no xdist by design — a stack is already 5-8 OS processes). A missing Redis/Postgres fails loudly at session start with the docker compose up -d hint. The shared Redis needs no modules — access control stores its records as plain hashes and counters.

Variant legs

Every stack is rendered through a plugin VARIANT triple, selected per pytest process and printed in the run header. An unknown value fails loudly, naming the valid set.

# the rq and celery backend legs — celery needs its broker
docker compose --profile celery up -d
TAI_E2E_BACKEND=rq     uv run --no-sync pytest -m "not backendless"
TAI_E2E_BACKEND=celery uv run --no-sync pytest -m "not backendless"

# the second identity/storage values, on the identity/storage-bearing suites
TAI_E2E_IDENTITY=fixture TAI_E2E_STORAGE=fixture uv run --no-sync pytest tests/redis_semantics tests/storage tests/harness

backendless marks a module whose stack runs no backend worker: it exercises no backend seam, so the non-default backend legs deselect it with -m "not backendless" instead of re-running it three times.

The checkpoint Redis (agents state)

The langgraph redis checkpoint/store provider needs the RediSearch + RedisJSON modules, which redis:7-alpine does not carry. A second, module-capable Redis is compose-gated behind the agents-redis profile and pointed at by TAI_E2E_CHECKPOINT_REDIS_URL (see .env.example); the specs that drive it skip loudly with the compose hint when it is unset, and session start fails loudly if the URL points at an image without the modules.

docker compose --profile agents-redis up -d

Bug classes

Layout

• src/tai42_e2e/ — the harness library (imported by tests, never by the SUT): boot engine (stack.py), process spawning (procs.py), Redis/PG/RabbitMQ admin (redisx.py, pg.py, rabbitx.py), the plugin-variant adapters (variants.py), manifest/env profile builders (manifests.py), the scripted LLM stub (llmstub.py), the net fixtures (netfixtures.py), the severable TCP relay the outage specs inject with (tcprelay.py), the single waiting primitive (waiting.py), and failure diagnostics (diagnostics.py).
• src/tai42_e2e_fixtures/ — SUT-SIDE modules the spawned server imports via its manifest: the probe tools (tools.py), the managed MCP server the fixture connector descriptors launch (managed_mcp_server.py), and the second value on each pluggable variant — the PG-backed identity provider (identity_provider.py), the storage backend (storage.py), and the monitoring backend (monitor_backend.py). The fixture connector descriptors themselves ride the manifest connectors field (built in tai42_e2e.manifests).
• tests/ — the suites by bug class, plus tests/harness/ self-tests.
• docs/adding-a-test.md — the 6-step recipe for a new feature's e2e test.

Metrics-dir isolation (the hard rule)

The harness NEVER sets PROMETHEUS_MULTIPROC_DIR in a child env — stamping it is the entrypoint's own job (activate_multiproc_env) and is exactly what the import-order tests verify. Per-stack (and per-replica) metrics dirs are controlled entirely via TMPDIR: MetricsSettings.prometheus_multiproc_dir defaults to <tempfile.gettempdir()>/tai42_prometheus, so pointing a process's TMPDIR at a per-run-family dir gives that family its own multiproc dir with the env var untouched. stack.py asserts the rule on every child env.

xfail policy

A test for a fix that has not yet landed in the workspace is marked xfail(strict=True) with the observed-behavior reason, so it flips loudly (xpass → failure) the moment the fix lands.

CI

The fleet e2e runs from the monorepo root workflow, .github/workflows/ci.yml, in two jobs gated by the changes filter (any member carrying an e2e trigger, or a change to a shared root, sets run_e2e):

• e2e — one matrix leg per triggered backend (arq, rq, celery), under TAI_E2E_FLEET=1 with redis identity and local storage. Services: redis:7-alpine (shared), postgres:16-alpine, and redis:8 — the module-capable checkpoint Redis the arq leg points TAI_E2E_CHECKPOINT_REDIS_URL at. The celery leg brings its RabbitMQ up from the compose celery profile; non-arq legs deselect -m "not backendless". The langfuse monitoring profile runs on the arq leg only when the run_monitoring gate is on (a change under plugins/monitoring-langfuse, core, or e2e, or a manual workflow_dispatch). When the private-repo token is present it also runs the marketplace suite (TAI_E2E_MARKETPLACE=1), installing the git-sourced tai42-marketplace and booting it as harness processes.
• ui-e2e — needs e2e; boots redis + postgres, clones tai-studio, builds the Studio SPA, and runs the Playwright chromium suite in ui/. When the run_browsers gate is on (a change under e2e/ui, or a manual workflow_dispatch) it also widens the matrix onto firefox + webkit. With the token it also clones the private tai-marketplace monorepo (its web/ workspace is the browse site) and runs the marketplace browser specs under TAI_E2E_MARKETPLACE=1.

The marketplace legs need the private tai42ai/tai-marketplace repo, so they run only when the RELEASE_PLEASE_TOKEN secret is available — a fork PR has no secrets, so those steps skip with a visible ::warning:: rather than failing. Other external consumers install this harness as the published tai42-e2e package and run their own suites against it. Any consumer booting the studio stack must also install the Studio reference plugin from the tai42ai/tai-studio repo (e2e/reference-plugin), which the studio profile manifest names but the harness does not carry as a runtime dependency.

Marketplace area (opt-in)

tests/marketplace/ only collects when TAI_E2E_MARKETPLACE=1; unset, it is skipped at collection like tests/monitoring/. When on, the harness installs the tai42-marketplace registry out-of-band from its pinned git source at boot and boots it — as a harness-managed process (the API server) on an isolated Postgres database: no extra compose profile; it rides the core docker compose up -d infra. The suite forges REAL fixture wheels (and one source tarball) from the in-repo fixtures/marketplace_plugins tree (TAI_E2E_MARKETPLACE_FIXTURES overrides it), served over a local package index, seeds them through the registry's real ingest pipeline, which publishes each version synchronously, and the skeleton REALLY pip-installs them into the one shared venv — so every lifecycle spec ends with its own uninstall and a session guard raises loudly on any leftover distribution (never auto-cleaning). The ui leg additionally pnpm-builds and serves the public tai-marketplace/web checkout and drives the Studio marketplace page and the public site over a browser.

docker compose up -d
TAI_E2E_MARKETPLACE=1 uv run --no-sync pytest tests/marketplace
# the browser legs (built Studio dist + the tai-marketplace/web checkout):
cd ui && TAI_E2E_MARKETPLACE=1 pnpm exec playwright test tests/marketplace.spec.ts tests/marketplace-web.spec.ts

TAI_E2E_MARKETPLACE_REF (unset by default) pins the registry to a specific tai-marketplace commit instead of the checked-in pin: the value must be a full 40-char lowercase-hex sha, the dedicated registry venv is keyed by it, and the out-of-band install resolves that commit. The fleet-e2e workflow sets it to the source sha under test when a tai-marketplace push drives the run, so the registry boots at the exact commit being validated. When it is set, a route-carrying leg whose registry cannot accept declared routes FAILS (rather than skips), naming whether the registry venv was absent (a harness ordering bug) or the dispatched ref regressed its tai42-contract floor. Reproduce a dispatched run locally:

TAI_E2E_MARKETPLACE_REF=<40-hex-sha> TAI_E2E_MARKETPLACE=1 uv run --no-sync pytest tests/marketplace