litmus — the CLI, and how a repo talks to a workspace

One file, cli/litmus.py, stdlib only. It is the whole client. Everything it does is an HTTP call you could make by hand — the CLI is a convenience, never a requirement.

Getting it (three shapes, one file)

The first two are the same bytes. The wheel does not contain a rendering of cli/litmus.py; it contains that file, renamed on the way into the wheel so it imports as litmus_ai_sdk (pyproject.toml, force-include). So the digest litmus-ai --version prints is the digest GET /litmus.py.sha256 publishes, a customer can check one against the other, and the probe brief the console lifts out of the served text cannot drift from the client the customer installs.

GET /litmus.py is deliberately unauthenticated: it serves client code, which holds no secrets. The token is what is protected, and it never appears here.

The name

litmus-ai-sdk on PyPI. Sidharth's call, 2026-09-15, after weighing litmus-sdk / litmus-ai-sdk / litmussdk / litmusai — and made because the alternative was worse: the probe prompt was telling a customer's agent to curl a script off a shared-workspace hostname built out of another customer's project name — which reads, to the customer running the command, as an unexplained third party inside their own install instructions. (The reasoning in full, with the names, is internal: DIRECTION.md §5, 2026-09-15/16. This file is the distribution's description on PyPI, so it names no customer.) A pip install line names nobody's infrastructure. It is not up for revisiting.

Two names he did not specify, which are just as permanent, and why they are what they are:

The collision, and why the console script is litmus-ai

This repo declares two distributions and they share a word:

• the root pyproject.toml — distribution litmus, package litmus/, console script litmus. The internal V1.1 product: run, grade, analyze, export, compare, world, validate, provision, fixture. Customers never install it and it is published nowhere.
• cli/pyproject.toml — distribution litmus-ai-sdk, module litmus_ai_sdk, console script litmus-ai. This file. What a customer installs.

If the second one also installed a script called litmus, then in any environment holding both — which is every Litmus engineer who reproduces a customer issue — litmus would be whichever was pip-installed most recently. That is not a theoretical hazard, because the two tools share the verbs export and compare with entirely different meanings, different flags and different outputs. litmus export would quietly do the other thing, succeed, and produce a wrong artifact. A name that can be shadowed is a name that will be.

So: one script, litmus-ai, which can neither shadow nor be shadowed, matches the distribution name the reader just typed, and fails loudly (command not found) for anyone following an older doc — which is the correct failure. Python entry points are per-distribution and per-environment; nothing about pip install litmus-ai-sdk can now reach the internal tool, in either direction.

Treat all three names as permanent: they go into customer prompts, agent instructions and support transcripts, and the cost of changing one later is every one of those.

The version — what the number means and when it moves

__version__ in cli/litmus.py is the only place the version is written; cli/pyproject.toml reads it, so the wheel on PyPI, the module a customer imports and the bytes the workspace serves at GET /litmus.py cannot claim different versions of the same file.

The scheme is MAJOR.MINOR.PATCH, and what it versions is the surface a customer meets — the verbs, their flags, and the briefs the client prints. Not the internals: this file is one module, and "an internal refactor" is not a category a customer can observe.

Every commit that changes the bytes of cli/litmus.py moves the number, in that same commit. Not at release time, not when someone decides a batch is worth shipping. The reason is the rule this repo already holds one product over: a fix to a cached derived layer that does not bump the layer's version has not shipped, it has only been written (DIRECTION §5 (61), (64), (73)), and the published wheel is exactly such a layer.

This rule was already written here, in prose, on 2026-09-16, and it was broken within three days. census landed in BRIEFS on 2026-09-19 with __version__ still reading 0.1.0 and nothing uploaded; a MedScout probe agent ran litmus-ai brief census, got usage: litmus brief <probe>, and found it before we did. A rule nobody can hold is a rule nobody applies — so it is a test now, not a paragraph. tests/test_published_client_contract.py reads the release off PyPI and fails when these bytes differ from it under the same number, and fails again when anything a customer is told to run is missing from what was actually published.

Current: 0.5.0 (MINOR, LIT-525) — a verb and two flags ADDED. target add|list|use|remove keeps named workspaces in ~/.litmus.json, each with its token and the token's kind as /v0/whoami named it; every verb takes --target <name> and prints, on stderr, which target and which kind of token it used. org export takes --out <file>, and org import reads a pipe, so org export --target prod … | org import --target dev … moves an org with no file on disk. Behaviour that changed in the same release: org export writes nothing unless the response is an org archive (status, content type, and a manifest.json of this format and this org, read before the first byte is written), and names the route a target lacks; org import refuses a file or pipe that is not one before sending it. init also saves its pair as the target default.

0.4.0 (MINOR, LIT-465; LIT-472 rides in it) — two verbs ADDED. pull --from datadog-llmobs fetches a datadog source's LLM Observability spans since its last pull and pushes them through the same convert, anonymise, census and batch path as push --from datadog-llmobs. Keys are read from DD_API_KEY, DD_APP_KEY and DD_SITE by name, and the span scope is read from the row. sources set-scope|set-wrapper sets what a pull reads and names the MCP wrapper's Datadog service. The push also prints a pipeline line now. In the same release (LIT-472), push takes a .pptx, .pdf or .docx as a document: the bytes go as base64 with format, the workspace converts them, and the push prints the slide, page or section count and the paragraph count. No new flag and no new dependency; a file over 20 MiB is refused before it is sent.

0.3.8 (PATCH, LIT-382) — mock --serve and --serve-http now advertise a write's row fields in tools/list (the manifest's param_schemas), as the workspace's MCP doors already did; before, the same bundle was served here as "a list of objects". Only the advertised schema changes: arguments still reach tools.py as sent.

0.3.2 — the PATCH number LIT-255 pinned for the hotfix, although it also ADDS a flag (--explain-drops), which the table above would call a MINOR; the next added verb or flag goes to 0.4.0 rather than compounding that. The fix: a root parent_id that is a SENTINEL (undefined, null, none, nil, 0 — Datadog LLM Observability writes the first one) is now read as "no parent" rather than as a dangling id, so an export whose roots say undefined converts instead of dropping whole (0.3.1 read 64,605 spans of a real export and emitted nothing). Drops now carry the SHAPE of the parent ids behind them — classes, lengths and counts, never an id — and a root may be inferred, and says so on the trace, in exactly one narrow case (LIT-255).

Publishing

.github/workflows/publish-cli.yml builds and uploads on a push to main that touches cli/**, and on manual dispatch. It authenticates to PyPI by trusted publishing (OIDC), so there is no API token in GitHub secrets, and it no-ops until the repo variable PYPI_PUBLISH_ARMED is set — merging it ahead of the PyPI-side setup is safe. It refuses to upload a version that is already on the index, so a cli/README.md-only change is a no-op rather than a failure.

By hand, if you ever need to — publishing is outward-facing and irreversible, a PyPI name and version can never be reused:

cd cli
uv build --out-dir dist/                       # or: python -m build
python -m twine upload dist/*                  # needs a token

twine reads TWINE_USERNAME (the literal __token__) and TWINE_PASSWORD — an env var name; the value is a PyPI API token that lives in the secret store and never in a file, a log, a commit or a URL. That token is what trusted publishing exists to retire: it is long-lived, it is copyable, and it is not scoped to a commit.

Editing litmus.py is a release, not a commit. The console's probe prompt tells every customer to hold litmus-ai --version against the digest their workspace publishes at GET /litmus.py.sha256, and that comparison is only true while the published wheel and the deployed file are the same bytes. A merge to main redeploys every workspace within minutes, so a change here that ships without a matching version bump and upload leaves the workspace serving bytes no installed client matches — and the console has already told the customer to compare two numbers that will not agree. So, in one commit: change litmus.py and bump __version__; the publish workflow does the upload on the merge. The prompt tells a customer what to do when the two differ (pip install -U litmus-ai-sdk, or the workspace curl, which is that workspace's own bytes by construction) — that is the recovery, not permission to let them drift.

The console-side wiring this unblocked is done: dashboard/lib/brief.ts renders pip install litmus-ai-sdk first, keeps the curl block as a named fallback for a machine with no index, and spells every command litmus-ai.

Verify the download. A reviewer refused to run this file because it arrived with "no checksum, no integrity guarantee" — correctly. The response carries an X-Litmus-SHA256 header, and GET /litmus.py.sha256 returns the digest on its own so it can be fetched by a different route than the code:

curl -sO <workspace>/litmus.py
curl -s <workspace>/litmus.py.sha256          # {"sha256": "…", "bytes": …}
shasum -a 256 litmus.py                       # compare

Until the workspace has TLS this detects tampering rather than preventing it — anyone able to rewrite the body can rewrite the header. The digest is worth having anyway (it pins exactly what you ran), and it is not a substitute for either TLS or, better, an installed package with a pinned version.

A CLI is not the non-technical surface, and pretending otherwise is how products get built for the wrong person. The non-technical path is the console (/console) and the dashboard: paste a URL, read a report. What makes the CLI usable by a non-expert is that they never type it — their coding agent does, after reading litmus brief probe. The human types English.

The loop

curl -sO http://<workspace>/litmus.py

# the token is a per-org API token from the console (Settings → Tokens, scope
# `push`); it pins the org, so `push` needs no --org. Never the service token.
python3 litmus.py init --url http://<workspace> --token litk_…

# 1. map a real tool surface (nothing leaves the machine yet)
python3 litmus.py probe --mcp "npx -y @acme/mcp-server"
python3 litmus.py probe --mcp "https://acme.example.com/mcp" --header "Authorization: Bearer …"
# hosted and OAuth-protected? Nothing to configure: the 401 is discovered, a
# browser opens once, you sign in as yourself. docs/ops/sign-in-to-a-hosted-mcp.md
python3 litmus.py probe --mcp "https://acme.example.com/mcp"

# 2. ship the evidence
python3 litmus.py push .litmus/sessions/<session>.mcp.jsonl

# 3. get a mock back — compile + mint, one live MCP URL out
python3 litmus.py serve acme .litmus/sessions/*.mcp.jsonl

# 4. the question the product rests on: is the mock like the real thing?
python3 litmus.py compare --real "npx -y @acme/mcp-server" --mock <that URL>

Installed, the same loop is litmus-ai init …, litmus-ai probe … and so on — identical bytes, identical verbs, just on $PATH. litmus-ai --version prints the version, the file's sha256 (compare it with GET /litmus.py.sha256) and the workspace it is pointed at; litmus-ai status is the one that asks the workspace for its build version, because that is a fact about a different machine and --version makes no network call.

More than one workspace: named targets

An operator holds two workspaces (production and dev) and two kinds of token (an org token, a workspace's service token). Rather than two pasted env vars per command, the client keeps them by name:

python3 litmus.py target add prod --url https://api.litmus.ai --token-env PROD_ADMIN_TOKEN
python3 litmus.py target add dev  --url <dev workspace>       --token-env DEV_SERVICE_TOKEN
python3 litmus.py target list            # name, url, token KIND, org — never the token
python3 litmus.py org export --target prod --org <slug> | python3 litmus.py org import --target dev --org <slug>

Which token kind each verb wants, where each is minted and where to keep it is one table: docs/ops/onboard-a-customer.md, "The three tokens, and which command wants which".

Steps 1 and 4 need no Litmus account for the real half — probe writes local files only. A workspace URL and token are needed from step 2 on. A per-org token opens exactly one org: push, and reading that org's evidence, jobs and pipeline back; used against another org it is refused with the mismatch named, and a revoked or expired one is refused with the reason. serve, shell and status still need the workspace's service token, which customers do not get.

The verbs

• probe --mcp <server cmd | url> — mechanical sweep of every tool: required args filled from the schema, ids harvested from earlier replies and reused, optionals both supplied and omitted, and a deliberate bad id at the end because error envelopes are part of the surface. --read-only skips anything that does not look like a read.
• call --mcp <…> <tool> --args '{…}' — one recorded call. This is the verb an agent drives, thinking between calls; that is why the protocol in litmus brief probe beats an unattended sweep. Calls append to the same session file.
• A server that answers 401 is signed in to, not given up on. probe and call implement the MCP authorization specification: the challenge is read, the protected-resource and authorization-server metadata are discovered, this client registers itself (RFC 7591) where the server allows it, and the person signs in once in a browser with PKCE and a loopback redirect. The token is reused after that and lives in ~/.litmus-mcp-auth.json at mode 600 — never in .litmus/, which is what push sends us. --header is unchanged and suppresses the flow (you supplied the credential); --client-id carries a client id from a vendor who does not register clients dynamically; --no-auth runs no flow at all, which is how a census records what the surface answers with NO credential. Failures name which of the five causes fired: docs/ops/sign-in-to-a-hosted-mcp.md.
• record --wrap gh,gt -- <command> — for surfaces that are CLIs, not MCP: shims wrap the real binaries and log argv/stdout/exit.
• push <file> — traces, tickets, workflow text, telemetry.
• serve <name> <sessions…> — compile through the replay gate and mint a world. A mock that cannot reproduce its own evidence refuses to serve, and says why.
• compare --real <…> --mock <…> — replay identical calls against both.
• shell --bundle <name-or-dir> --mint — materialize a world locally and PATH-shadow its fake binaries, so an agent runs against the mock unmodified. A name rather than a path is fetched from the workspace and cached in .litmus/bundles/, digest-checked. This is the path for agents whose surface is CLIs rather than MCP: litmus shell --bundle <bundle-id> --mint gives you working linearis, huginn, gh and gt on a machine where none of them are installed, and logs every invocation into the world.
• brief probe — the two-phase protocol, as instructions for someone else's agent: phase 1 maps the surface (schemas in and out, auth, errors, refusals, which tools write), phase 2 plays out whole jobs so the corpus carries user turns a task can quote. Intelligence flows down; evidence flows up. The fidelity 1.00 on Litmus's battery was measured on brief v1, which was phase 1 alone; v2 does not inherit it.
• export --org <org> --surface <surface> — assemble the thing a customer receives (a customer PRD's technical requirement 7): the pipeline's suite and mock byte-for-byte, the runner vendored as source with its dependencies pinned by hash, this file with its digest in the README, a config that names secrets by env var only, a manifest of every pin, and a run.sh that runs cells, grades them and renders a scorecard — with no workspace in the loop. It assembles; it authors nothing, and a pin the pipeline did not produce is null in the manifest with the reason. Reproducible: two exports differ only in the timestamp.
• target add|list|use|remove — named workspaces, each with its token and the token's kind; --target <name> on any verb picks one, and the verb's first line on stderr names the target and the kind it used. target list never prints a token.
• org export|import — one org between two workspaces as one archive. The export writes nothing unless what came back IS an archive, and names the route a target lacks; --out renames into place only a whole archive.
• scorecard <results-dir> — render a scorecard from stored runs alone: per cell n, the five terminal states with infra_failure/invalid shown and excluded, the key's facts one by one, each detector's named findings, and a tier and a path on every number. No bare ratio.

How compare grades

Byte-identity is the wrong bar: a simulator that mints its own ids is still faithful if the shape and the outcome hold. So each replayed call lands in one of five buckets, worst-first in the report:

Plus a catalog diff: tools the mock invented, and tools it missed. --push files the whole report in the workspace as evidence.

One thing compare cannot do for you

If the replayed calls mutate, the two sides are not starting from the same place. The mock gets a freshly minted world; the real surface is wherever your last probe left it — cart already full, order already placed, the id you are fetching now existing. Divergence then measures state drift, not fidelity, and it will look like the mock is wrong when it is not.

So for a trace with mutations in it, one of these has to be true:

• the real surface is reset between the probe and the comparison (a fresh staging tenant, a seeded test account, a teardown script), or
• you compare on a read-only subset, or
• you read the result as a state diff and not a fidelity number.

There is no way for us to reset your system from here, so this is a fact about the measurement rather than a bug we can close. compare reports what it saw; knowing which world the real side was in is your half of the contract.