Skip to main content

introspection-harbor

Harbor installed-agent adapter for Introspection Recipes. Install it through introspection setup --target harbor, then run a task through:

introspection eval run --runner harbor --path evals/refund-task

Select an inherited Recipe agent variant by its declared YAML name:

introspection eval run --runner harbor --path evals/refund-task --agent agent2

The selected agent YAML is the only source of the evaluated model; the CLI and adapter do not apply a separate model override.

The CLI records that declared model as candidate.model in the run spec and run.json. The native adapter accepts it as candidate metadata and submits only the prompt to the task. The Runtime loads the selected agent from the candidate Recipe commit and uses its declared model. The CLI checks the model observed in the returned trajectory against candidate.model and records it as result.observed_model in run.json.

Harbor's own agents

A prefixed agent names one of Harbor's agents instead of a Recipe agent: harbor:oracle runs the task's solution/ through the environment's exec and harbor:nop sends no turn, both without a Recipe agent or a model. Harbor is invoked with that agent in place of the Recipe adapter, and on the Introspection environment the task keeps the Recipe's default agent while the solution runs through the task's commands. The prefix is the runner's name, so it is refused with a runner that has no agents of its own, and -a after -- stays refused: the common option is the one way to choose.

introspection eval run --runner harbor --path evals/refund-task --agent harbor:oracle

Kept outputs

When the agent finishes, before the verifier runs, the adapter copies what a verifier reads into /logs/agent/kept-outputs/: the Runtime workspace's files/ and outputs/ (under $INTROSPECTION_WORKSPACE_DIR, /workspace by default) as workspace/files/ and workspace/outputs/, Pi's sessions as session/, and each fake's final state as fakes/<id>.json for every task MCP server and every API binding of the Recipe's .pi/api.local.json under $INTROSPECTION_FAKES_URL. Harbor's default verifier shares the agent's sandbox, which is gone once the trial ends, so this copy is what is left of the agent's work. introspection eval run moves it to eval-runs/<run-id>/cases/<case>/. The copy is best effort and never fails a trial; a trial Harbor drives through a simulated user keeps nothing yet.

Input files

A task gives the agent files the way a user's upload does. Put each file in the task's environment/files/, and list it in the <files> block at the end of the instruction as /workspace/files/<name>, the block the Runtime appends to the message that first carries an upload:

Review this SOW.

<files>
<file path="/workspace/files/sow.docx" />
</files>

introspection eval add writes both from a stored conversation. The agent finds each file at /workspace/files/<name>.

The block goes where production put it: on the turn the files were uploaded. A task made from a later turn of that conversation has it in a user message of environment/history.json, not in instruction.md. Setup checks that the blocks of history.json and the instructions together name exactly the files in environment/files/, and fails before the agent starts otherwise, naming each file the blocks miss and each listed path with no file, with the step (or history.json) that lists it. A task with neither files nor a block runs as before.

A file reaches /workspace/files/ when production's upload would: at setup when history.json or the first step announces it, and otherwise right before the first step of a multi-step task whose instruction announces it. Until then the sandbox has no such file, as production has none before the turn an upload arrives on. Harbor's own agents (--agent harbor:oracle) place nothing per step, so on the introspection environment their task carries every file from setup.

How the files get there depends on the adapter:

  • runtime: Pi runs in production's workspace, on the Runtime image or a task's own image: its working directory is /workspace, INTROSPECTION_WORKSPACE_DIR=/workspace, and files/ and outputs/ exist and are the agent user's, so a Recipe tool reads /workspace/files/<name> as inside its workspace. In Docker and E2B the adapter creates them, uploads the setup files and gives them to the agent's user; on the introspection environment the task's runtime downloads the setup files. A later step's file is uploaded before that step either way. The first user message is instruction.md byte for byte, block included, as a plain message rather than an attached file, on every step and whether run sends it or a simulated user sends it over the ACP bridge (an instruction starting with @ or -, or above 100 KB, goes to Pi on stdin, which Pi trims).
  • native: the introspection environment uploads the files and attaches the setup ones to the trial's task, which the Runtime downloads at boot and announces itself, as it does for an upload; a later step's files go on that step's runs.create, as an upload on a later turn does. The adapter sends instruction.md without its block, so it is not announced twice. The uploads are deleted when the trial ends.

The directory is flat: no subdirectories or symlinks, at most 100 files of 25 MB each, the platform's limits on a task's files. Names starting with a dot are skipped.

Multi-step tasks

The steps of a multi-step task are turns of one conversation: a later step sees everything the earlier steps said and did.

  • --fresh-steps starts a new conversation at every step instead. The native adapter refuses it: its steps are runs on the trial's one task, which holds one conversation.
  • Harbor's own agents (harbor:oracle, harbor:nop) and production replays always start fresh.
  • A task that replays environment/history.json refuses to continue into a later step, since its session is hydrated afresh for each run; give it one step, or run it with --fresh-steps.
  • With harbor run directly, steps are fresh unless you pass --resume-trajectory.

A verifier reads the conversation from /logs/agent/trajectory.json. On a step that continues the conversation it covers every turn so far, so a later step can be graded against what an earlier one said. pi.jsonl covers the current step only, and a step's reported tokens and cost are that step's alone.

The trajectory is the platform's own ATIF projection. During each Pi run the Recipes extension writes the session as the GenAI spans the platform stores (genai-spans.json), and introspection-sdk projects those onto ATIF with the code behind the platform's ATIF conversation export. A trial and a production conversation therefore produce the same trajectory for the same turns. A step's usage comes from the spans that start after its Pi run was launched, and its input tokens include cached tokens, as ATIF defines prompt_tokens.

The spans are the trajectory's only source. The adapter names the file in PI_RECIPE_GENAI_SPANS_FILE and the extension replaces it after every turn, so a run that Harbor stops on its timeout still has the trajectory of the turns it finished. A trial whose Recipes extension predates the variable (an older Runtime image, or a CLI whose introspection setup installs an older Recipes release) fails after the agent run with an error that names the missing genai-spans.json. A run that made no model call has no trajectory.

Retrying infrastructure failures

--retry-errors <n> reruns, up to n times within the same run and the same Harbor job, the trials that failed on infrastructure rather than on the agent:

introspection eval run --runner harbor --path evals --retry-errors 2

A trial is rerun when every exception Harbor recorded for it, on the trial or on one of its steps, is one of these:

  • AgentTimeoutError with zero tokens used: Pi got no model response.
  • DataPlaneError for a rejected or expired login (401, "login has expired"), a 408, 429 or 5xx response, or a sandbox that never accepted commands.
  • A transport error of the adapter's HTTP client, such as ConnectError or ReadTimeout.
  • EnvironmentStartTimeoutError, AgentSetupTimeoutError, VerifierTimeoutError, AddTestsDirError, DownloadVerifierDirError and CancelledError.

Everything else stays as it is: an agent that timed out after using tokens or whose token usage is unknown, a non-zero agent exit, a missing or unreadable reward, a request the Data Plane refused (a Recipe that fails validation, a missing scope, an unknown Runtime), a sidecar image that fails to build, and any trial that completed with a low score.

The rerun is harbor job resume on the job directory, after the failed trials' directories are removed from it. run.json lists each rerun case under result.retries with its attempts and the kind and message of every discarded trial; a case that is not listed took one attempt. case is the id the case has in result.cases, which holds the last attempt only. The flag cannot be combined with --from-conversation. Harbor's own --max-retries after -- still works and selects by exception type alone.

Local Docker on the Runtime image

With no --env and INTROSPECTION_HARBOR_ENVIRONMENT unset, introspection eval run runs Harbor's local Docker environment through introspection_harbor.docker_environment:IntrospectionDockerEnvironment. A task that names its own main (an environment/Dockerfile, [environment] docker_image, or an image or build on the compose file's main) runs that image. Any other task, such as the one introspection eval add --runner harbor writes, runs main in the Runtime image, as a plane trial does: the environment starts introspection-runtime-agent:latest. INTROSPECTION_HARBOR_RUNTIME_IMAGE names another image for main.

Either way the environment prepares the Recipe being measured at /opt/introspection/recipe before the agent starts. That is the working tree without its eval suites, cases and eval code, with its production dependencies installed with pnpm from its pnpm-lock.yaml and its peers linked. As runtime-agent does, the install is not frozen: its peers are linked rather than installed, which a lockfile recorded with autoInstallPeers: true contradicts. As on the plane, a Recipe with dependencies and no pnpm-lock.yaml is refused. In the Runtime image the dependencies come from the image's warm store, the peers from its runtime and the runtime link goes beside the Recipe. In a task's own image the dependencies come from the network (with corepack's pnpm when the image has none), the peers from the packages introspection setup installed (or from Pi's own dependencies, for a package only Pi requires, such as typebox), and there is no runtime link; a peer the image lacks is refused with the toolchain fix. A Recipe in a subdirectory of its workspace gets the workspace's .introspection/ manifest for it, with path: ., so introspection local finds it. --env docker runs Harbor's own Docker environment, which needs the task to name its image and prepares no Recipe.

A task's own image

A task that needs its own filesystem (a pinned SWE world, a customer base) names its image and installs the toolchain the runtime adapter runs on, on Node 24:

FROM node:24-slim
RUN npm i -g @introspection-ai/cli@<version> && introspection setup --yes

The adapter checks the image's CLI is 0.78.0 or later and that its pi lists the Recipes extension, and refuses the trial naming that line otherwise (CliTooOldError, ToolchainMissingError). In such an image introspection local finds no Runtime sandbox and runs the Pi introspection setup installed, with the Recipe. tests/test_runtime_docker.py runs such a task in a real Docker trial, with a fake model in a sidecar (HARBOR_RUNTIME_DOCKER=1 uv run python -m unittest tests.test_runtime_docker; it installs from npm, and HARBOR_RUNTIME_DOCKER_CLI_VERSION picks the CLI).

The default runtime adapter expects that layout: the Recipe under /opt/introspection/recipe and the runtime under /usr/local/lib/introspection-runtime, with an Introspection CLI no older than the adapter's floor. A plane's Runtime images are private, but make docker-build-runtime-agent in introspection-cloud builds the base they are built on as introspection-runtime-agent:latest. A task that brings its own image can start from it too:

FROM introspection-runtime-agent:latest
COPY --chown=appuser:appuser . /opt/introspection/recipe

The Recipe's .pi/mcp.local.json binds a fake MCP server the compose file serves as a sidecar. The adapter writes it from the task's [[environment.mcp_servers]], where name is the server id the Recipe declares and the transport is streamable-http. A task that declares none gets the file seeded from .pi/mcp.local.example.json when it is absent.

The Recipe's .pi/api.local.json binds its API fakes the same way. When the Recipe has fakes/api/, the adapter rewrites the file before Pi starts: each host a fakes/api/<id>/manifest.json lists goes to ${INTROSPECTION_FAKES_URL:-http://fakes:7353}/<id>, the trial's fakes service, which must serve the API fakes too (@introspection-ai/eval-simulate 0.9.0 or later, with @introspection-ai/recipes 0.52.0 or later in the image). A manifest without hosts, or a host two fakes list, fails the trial's setup. A Recipe without fakes/api/ keeps a committed file as it is. The native adapter binds the API fakes too, with or without task MCP servers. A task's own image takes the runtime adapter too, once it has the toolchain (see A task's own image).

Where the Recipe is, for a verifier

A verifier's environment carries INTROSPECTION_RECIPE_DIR, the directory the Recipe ran from in that trial, so a check that reads the Recipe needs no path of its own. Pass it to RewardKit as the workspace that file checks such as file_exists resolve against and an isolated agent judge runs in:

uvx --from harbor-rewardkit rewardkit /tests --workspace "$INTROSPECTION_RECIPE_DIR"
Adapter Environment INTROSPECTION_RECIPE_DIR
runtime any (docker, e2b, introspection) /opt/introspection/recipe
native introspection /opt/introspection/recipe
none of these (harbor:oracle, say) introspection /opt/introspection/recipe

The adapter sets it on the environment in setup, so it reaches every later command of the trial, the verifier's included, on every environment. A value the task sets in [verifier.env] takes precedence. On other environments a trial without an Introspection adapter gets none. The meaning is the one introspection judge run reads: the Recipe's root.

Reading the trajectory in a verifier

A task's verifier finds the conversation at /logs/agent/trajectory.json. introspection_harbor.verifier reads it without the check knowing the ATIF layout:

from introspection_harbor.verifier import (
    final_reply,
    load_trajectory,
    server_calls,
    tool_calls,
)

trajectory = load_trajectory()
final_reply(trajectory)  # the agent's last non-empty message
tool_calls(
    trajectory
)  # [{"name", "arguments", "result", "is_error"}], answered calls only
server_calls(tool_calls(trajectory), "attio")  # one MCP server's calls

server_calls(calls, server, tools=None) keeps one MCP server's calls under the server's own tool names. Pi names an MCP tool mcp_<server>_<tool>, with characters outside [A-Za-z0-9_-] replaced, and may cut it short and append _<10 hex>, a hash of the name. Given the server's tools, such as its answers from load_answers below, each call is named by the tool whose Pi name it carries, and a call to no listed tool is left out; without them, the hash is stripped, which recovers only a short name Pi did not have to replace characters of. Each is {"name", "args", "isError", "response"}, the keys of serverCalls in @introspection-ai/vitest-evals, so a check ports between the two runners unchanged, plus structured, the result's structuredContent, when the call has it. A trajectory carries structuredContent only where the result's ATIF extra holds it as structured_content, and the GenAI spans it is projected from do not record it yet, so for now no call has it.

Add the package to the verifier's environment, for example uvx --from harbor-rewardkit --with introspection-harbor rewardkit /tests.

Checking a Recipe's MCP fakes

A Recipe's fakes run in one sidecar of the trial, on @introspection-ai/eval-simulate, which serves each fake's state at /<id>/admin/state and the state it started with at /<id>/admin/initial-state. introspection_harbor.fakes reads them and holds the two checks every case of a Recipe may share. Each returns what it found, so a criterion passes when it is empty:

from pathlib import Path

from rewardkit import criterion

from introspection_harbor.fakes import load_manifests, state_changes, write_attempts
from introspection_harbor.verifier import load_trajectory, tool_calls

# Each fake's manifest.json, which the sidecar writes there under EVAL_SIMULATE_MANIFEST_DIR.
MANIFESTS = load_manifests(Path("/opt/fakes-verifier/mcp"))
CRM = {"attio": MANIFESTS["attio"]}


@criterion(
    description="No CRM record, email or send attempt differs from the world the trial started in"
)
def nothing_changed(workspace: Path) -> bool:
    return not state_changes({fake: MANIFESTS[fake] for fake in ("attio", "resend")})


@criterion(description="No CRM tool that writes was called")
def no_crm_write_attempted(workspace: Path) -> bool:
    return not write_attempts(tool_calls(load_trajectory()), CRM)
Function Returns
fake_state(id, url=None), fake_initial_state(id, url=None) A fake's state now, and when the trial started
load_manifests(directory) {id: manifest} from <directory>/<id>/manifest.json; raises when the directory is missing or holds no manifest
state_changes(manifests, url=None) ["<id>.<entity>", ...]: every entity some tool of these fakes writes whose state differs from the initial one; empty when no state changed
write_attempts(calls, manifests) The calls to a tool the write maps say writes, each with its server (fake id) and its contract tool name; empty when no write was attempted

The sidecar's base URL is url, else the INTROSPECTION_FAKES_URL environment variable, else http://fakes:7353, the trial's fakes service. It is reached without the trial's proxy.

Every check raises on an empty set of manifests rather than passing, and compares states as JSON, so false and 0 differ.

  • "No state changed" reads state. It compares the entities some tool of the fake writes, which include outbound effects such as sent emails and the send attempts a mail fake rejected. A rejected write or a no-op call is not a change; a change undone later in the trial is not seen, so a case that forbids a temporary change records each transition in the fake's state.
  • "No write attempted" reads the trajectory. It counts a call to a write tool whether or not the call changed anything, so name and score it as an attempt check, beside the state check rather than instead of it. A call is matched by the name Pi gives an MCP tool, from the fake's id (its directory under fakes/mcp/, which is the server id the Recipe declares) and the tool's name in its contract; readOnlyHint plays no part.

Reading the fakes' answers

introspection_harbor.answers reads a tool's successful answer as @introspection-ai/eval-simulate/mcp/answers does: decoded by its format (json, toon, prefixed_json, or text) and validated against its JSON Schema. eval-simulate serve writes each fake's answers beside its manifest, as mcp/<id>/answers.json under EVAL_SIMULATE_MANIFEST_DIR. TOON is read with toon-format, a dependency of this package, so a task's test.sh needs no --with toon-format.

from introspection_harbor.answers import load_answers, read_result

ANSWERS = load_answers(Path("/opt/fakes-verifier/mcp"))

for call in server_calls(tool_calls(load_trajectory()), "attio", ANSWERS["attio"]):
    if call["isError"]:
        continue
    read = read_result(
        ANSWERS["attio"][call["name"]], call["response"], call.get("structured")
    )
    if not read["ok"]:
        raise ValueError(
            f"attio answered {call['name']} outside its contract: {read['error']}"
        )
    rows = read["value"]
Function Returns
load_answers(directory) {id: {tool: answer}} from <directory>/<id>/answers.json; raises when the directory is missing or holds no answers
read_answer(answer, text) {"ok": True, "value"}, or {"ok": False, "error", "issues"} for text not in the answer's format or a value outside its schema; each issue is {"path", "keyword", "message"}
read_result(answer, text, structured=None) The same, read where the answer is: the structured content of a tool that states an outputSchema ("structured": true in its answer), whose text is ignored; the text of any other. Without structured content such a tool's read fails
decode(answer, text) The decoded value, unchecked; raises ValueError for text not in the format

An answer that fails to read is the fake's fault, not the agent's, so treat it as an invalid measurement rather than a failed criterion. Both packages' tests run the cases in packages/testdata/fakes/answers.json, so the two read an answer alike.

@introspection-ai/eval-simulate/mcp/verify holds the same checks for a vitest eval (changedEntities, writeAttempts), and both packages' tests run the cases in this repository's packages/testdata/fakes/cases.json, so the two runners score a trial alike.

A judge in a verifier

judge_reward(name) grades the trial with one of the Recipe's online judges, the YAML definition in judges/ whose name is name, and returns the verdict as one RewardKit reward. A case reuses the judge the platform runs on live conversations instead of restating it as a RewardKit [judge] TOML:

from introspection_harbor.verifier import judge_reward, write_rewards

write_rewards({"answers": judge_reward("answers-from-the-crm")})

It reads the definition from $INTROSPECTION_RECIPE_DIR/judges/ (a direct .yaml or .yml child; its filename means nothing) and the conversation from /logs/agent/trajectory.json. Pass recipe_dir= or trajectory_path= to read them elsewhere. The request is the one introspection judges evaluate sends for that definition and conversation: the definition's instructions and the engine's output requirement, the trajectory as the JudgeInput recipeJudge builds, the on gate, and the llm.request settings, on the chat-completions or the Responses API as the CLI chooses. The model is called through LiteLLM, a dependency of this package that harbor-rewardkit installs too; the CLI never runs in the trial. Tests on the fixtures in packages/testdata/judges/ hold this package and the CLI's judge engine to the same request and verdict.

Verdict score Criterion (named after the judge)
pass 1.0 value: 1.0, the judge's reasoning
fail 0.0 value: 0.0, the judge's reasoning
not_applicable, or a closed on gate None optional: true and an error saying why

The reward also carries judge, with the definition's name, its definition_hash (the SHA-256 of the file, as judges evaluate reports it) and its model and provider, and usage when the model was called. The CLI counts it as model-graded, copies the judge's name and hash to its criterion, and introspection eval compare names a criterion whose judge's hash differs between the two runs. Use the reward alone or as a component of a group. write_rewards(rewards, directory=Path("/logs/verifier")) writes reward.json and reward-details.json, leaving a dimension that did not apply out of the scores. An unknown name, a name two files declare, a missing key or a reply the engine would refuse raises JudgeError.

The provider key

The definition's provider names the key: OPENAI_API_KEY for openai, OPENROUTER_API_KEY for openrouter, ANTHROPIC_API_KEY for anthropic, or llm.local.api_key_env with llm.local.base_url. A RewardKit [judge] reads the same variables.

  • On the Introspection environment the verifier already holds the child task's session locator in that variable, and its provider base URLs point at managed egress, which swaps the locator for the credential (A verifier that calls a model). Nothing to configure.

  • On local Docker Harbor gives the verifier only the task's [verifier.env], the job's verifier env and the environment's own variables. Name the key in the task, filled from the environment Harbor runs in:

    [verifier.env]
    OPENAI_API_KEY = "${OPENAI_API_KEY:-}"
    

    and export it before introspection eval run. The task carries no secret, the job config eval run writes carries none, and Harbor scrubs the value from the logs it copies out of the trial. Harbor asks before a job loads a host variable this way. eval run answers it with Harbor's --yes only when every variable the selected tasks read through ${VAR} in [verifier.env] or [environment.env] is a provider key: one of MANAGED_PROVIDER_KEYS in base.py (ANTHROPIC_API_KEY, GEMINI_API_KEY, GROQ_API_KEY, MISTRAL_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY, XAI_API_KEY) or a judge's llm.local.api_key_env. Any other variable, or a task.toml it cannot parse, leaves the question to Harbor, with one line naming the variables. It never answers when the arguments after -- publish the job (--upload, --launch, --share), rerun earlier trials (--diff) or answer it themselves (--yes). Under harbor run directly, answer it or pass --yes. :- keeps a run with the variable unset from failing before it starts: judge_reward then refuses the empty key by name.

  • On E2B without managed egress the same line works: Harbor resolves it on the host before it runs the verifier, in whichever environment.

tests/test_judge_docker.py runs such a task in a real Docker trial, with a fake model inside it that checks the key it was sent (HARBOR_JUDGE_DOCKER=1 uv run python -m unittest tests.test_judge_docker).

Adapters, one launch

introspection eval run selects introspection_harbor.agent:IntrospectionRecipeAgent; INTROSPECTION_HARBOR_ADAPTER decides which adapter that resolves to. The runtime adapter's shared base, base.py, holds the run spec, the Pi invocation, the captured pi.jsonl, and the ATIF trajectory. It starts every step, ACP included, through introspection local. In the Runtime image that detects the Runtime sandbox and runs the image's Pi with the provider extension and preloads; in a task's own image with the toolchain installed it finds no sandbox and runs the Pi on PATH that introspection setup installed. native launches no Pi: each step is a run on the trial's evaluation task, so the task carries the prompt and the platform stores the conversation a judge reads.

INTROSPECTION_HARBOR_ADAPTER Trial image What install does How a step starts Pi Evidence
runtime (default) the Runtime image, or any image with the CLI and introspection setup run (a pinned SWE world, a customer base) checks the image's CLI is 0.78.0 or later and that it has Pi with the Recipes extension; downloads nothing introspection local, which runs the image's Pi in the Runtime sandbox, or the Pi on PATH outside one pi.jsonl in the trial
native the Runtime image, on the introspection environment nothing; each step is POST /v1/tasks/{id}/runs none: the platform runs the step the task's conversation, read back once ingest settles

The bootstrap adapter is gone: INTROSPECTION_HARBOR_ADAPTER=bootstrap fails like any unknown value, naming the fix. A task that used it on a foreign image adds the setup line to that image instead (see A task's own image) and leaves the variable unset. Node 24 is the image's to provide.

Committed candidates through an existing Runtime

Use introspection eval run --runner harbor --runtime <slug-or-id> --commit-sha <full-pushed-sha> --path <recipe-relative-suite> --agent <name>. The CLI fetches the exact SHA from the Runtime's registered Recipe repository using the existing Git credential helper and checks it out temporarily for Harbor suite discovery. It keeps the Runtime selected by --runtime; every trial sends that Runtime ID and the candidate SHA to the Task API. Neither a local checkout nor a branch flag is required. Operator forwards --commit-sha to this same command. Naming IntrospectionEnvironment by hand through --env without --commit-sha sends the local HEAD as the commit to materialize, so the CLI refuses a dirty working tree or a commit on no remote branch: the plane evaluates what was pushed, never the working tree. Committed-run results default to the invoking directory's eval-runs/ (or INTROSPECTION_EVAL_RUNS_DIR); they survive checkout cleanup. Any explicit Harbor --jobs-dir must be an absolute path outside the temporary checkout.

The Task API resolves that Runtime's trusted RuntimeSpawnContext and replaces only its Recipe commit for the eval Task. Repository, subpath, Runtime identity, bindings, resources, and image policy remain server-owned. The trial reuses the selected Runtime's published image. Runtime-agent stages the exact candidate checkout in the writable Task workspace, clears the baked Recipe directory's contents, copies the selected Recipe subpath into that existing directory, and reruns normal Recipe setup. The disposable sandbox keeps no backup of the baked source. There is no second workspace repository, manual token handling, uploaded checkout helper, candidate archive, or candidate bake. The pushed commit does not need its own Runtime-version record. A checkout, replacement, materialization, or dependency setup error is infrastructure/setup failure, not an agent regression.

Run the focused adapter checks with:

uv run python -m unittest tests.test_task_environment

These prove server-derived candidate selection, committed provenance, and the run environment without starting a platform Task or calling a model. The Harbor package must be released and installed in Operator before the production wrapper selects this adapter; the local Operator build already consumes the CLI checkout's Harbor wheel.

Operator also sets INTROSPECTION_EVAL_RUNS_DIR to its Task outputs eval-runs/ directory. The CLI writes native artifacts and run.json there directly, so their recorded paths survive detached-checkout cleanup. It must be an absolute path. introspection eval view honors the same override. Standalone CLI runs still default to Recipe-local eval-runs/; generated results are never included in committed candidate transport.

The Introspection environment

The adapters above choose how the agent reaches the trial. The environment chooses where the trial runs, and IntrospectionEnvironment runs it in an Introspection evaluation task:

introspection eval run --runner harbor --path evals/refund-task -- \
  --env introspection_harbor.task_environment:IntrospectionEnvironment

start creates a kind: eval task on the Recipe's Runtime, tagged harbor, with commands: true and the task's own docker-compose.yaml as its compose. Everything Harbor asks of the environment afterwards — exec, upload_file, download_file, and the healthchecks and directory helpers the base class derives from exec — becomes a row on POST /v1/tasks/{id}/commands that the task workflow runs in that sandbox.

The point of doing it on this axis: Harbor keeps the trial. The verifier runs through the same channel, in the same sandbox the agent worked in, so a task's own tests/test.sh produces the reward. The alternative — creating a platform task from an agent adapter and leaving the Harbor environment idle — reads the Recipe's judges back instead, which measures a Recipe but cannot score somebody else's benchmark. That shape was prototyped and withdrawn.

A verifier that calls a model

A judge script in tests/test.sh that uses the OpenAI or Anthropic SDK, or LiteLLM, works unchanged. Its calls use the sandbox's managed credential.

A provider key in [verifier.env] is dropped here, so the OPENAI_API_KEY = "${OPENAI_API_KEY:-}" a task declares for local Docker trials (A judge in a verifier) does no harm. Other providers are not supported for judges yet.

main is the agent slot and the Runtime image fills it — the agent base plus the Recipe, either baked or materialized from git — so the eval logic belongs in the Recipe repository beside the Recipe it measures. A main service therefore carries only volumes and depends_on, and a task that declares its own image: or build: there is refused by that key rather than trimmed: dropping it would run the trial in an image the task did not ask for and report the number as though it had. The same goes for an environment/Dockerfile that no sidecar builds from — for a task with no compose file at all, that is the only signal it was written for an environment that builds its own main. Every other service is a sidecar, and a sidecar may only name an image the plane produced — so a build: context is packed, uploaded and built through POST /v1/images, and an image: is mirrored, which requires it to be digest-pinned. Every task sends a compose, even one with no file of its own: Harbor's /logs and /tests sit directly under /, which the sandbox user cannot create, so they are mounted as named volumes the platform renders as emptyDir under the pod's fsGroup. _reset_dirs_user is overridden to the sandbox user for the same reason — asking for root produced a request the sandbox silently ignored and a mkdir that failed as if the trial had.

Credentials resolve as the CLI's own Data-Plane commands do: INTROSPECTION_DP_URL and INTROSPECTION_TOKEN first, then the profile introspection login wrote. Resolving a Runtime slug to its active version is a Control-Plane read, so a run that selects the Runtime by slug, or with --commit-sha, needs a login with a project selected. A run pinned to a Runtime version does not (below).

introspection eval run renews the stored login for as long as Harbor runs, so a run may outlive the access token it started with: the CLI refreshes the profile once the token has five minutes left, and a trial whose token is rejected reads the profile again and repeats the request once. An INTROSPECTION_TOKEN is used as given and is never renewed, and neither is the login under harbor run invoked directly.

A browser login is not sufficient on its own. A trial needs tasks:eval to create the task, images:write to build a sidecar and tasks:exec to run a command, all of which are in the evals capability — and a CLI login carries runtimes and reviews only. Set INTROSPECTION_TOKEN to a project API key minted with both evals and runtimes: the second is not redundant, since uploading a sidecar's build context goes through the Files API and files:write is in the runtimes bundle rather than evals. With a login alone the Control-Plane resolution succeeds and the task create then 403s.

Under a service credential

A trial pinned to a Runtime version makes no Control-Plane request, so a service credential alone runs it, with no person's login: a service account's Data-Plane token (an Application's client_credentials token for the project, its allowed_scopes naming evals and runtimes) or a project API key with those capabilities. A CI job holding either runs it:

INTROSPECTION_DP_URL=https://api.<plane> \
INTROSPECTION_TOKEN=<service account token or project API key, with evals and runtimes> \
INTROSPECTION_HARBOR_ENVIRONMENT=introspection_harbor.task_environment:IntrospectionEnvironment \
introspection eval run --runner harbor --path evals/refund-task \
  --runtime support-agent --runtime-version <runtime version UUID>

--runtime names the Recipe's local Runtime manifest by slug and --runtime-version pins the platform Runtime version the trial starts on. The trial runs that version as built, its image and the Recipe commit it was built from: the trial body names no recipe_git_commit_sha, so the plane checks out the version's own commit, and the local checkout need not be clean or pushed. run.json records the version as candidate.runtime and leaves recipe_git_commit_sha and recipe_git_dirty null, since resolving the built commit would take the Control-Plane request this run never makes. To evaluate a different commit on a Runtime, use --commit-sha instead. Redeploying the Runtime makes a new version, which the caller then pins.

Variable What it changes
INTROSPECTION_HARBOR_TASK_ENVIRONMENT the Runtime's environment lane for version resolution (default production)
INTROSPECTION_HARBOR_SANDBOX_PROVIDER pins the task to a sandbox provider
INTROSPECTION_HARBOR_TRIAL_TIMEOUT_SECONDS the sandbox's wall-clock ceiling, as the task's idle_timeout_seconds (default 3600)
INTROSPECTION_HARBOR_START_TIMEOUT_SECONDS how long to wait for the sandbox to accept commands, including queueing (default 3600)
INTROSPECTION_HARBOR_IMAGE_TIMEOUT_SECONDS how long to wait for a sidecar image to build (default 1800)
INTROSPECTION_HARBOR_EXEC_TIMEOUT_SECONDS the default per-command bound (default 900)

Timeouts, admission and fan-out

idle_timeout_seconds is the trial's wall-clock ceiling, not a grace period before the first command. Commands reach the sandbox through pods/exec, which the runtime agent never observes, so the idle seconds it reports climb for the whole trial however busy Harbor is. That is deliberate: it is also what releases the sandbox of a trial that has wedged. Size INTROSPECTION_HARBOR_TRIAL_TIMEOUT_SECONDS to the longest trial you expect, not to the pause before Harbor starts work.

A trial cannot exceed two hours. The plane clamps idle_timeout_seconds to the task's own timeout_seconds, and that field is not on POST /v1/tasks — it sits on the internal workflow request at a fixed 7200s — so no caller can raise it. Configuring a longer budget is refused at startup rather than accepted and halved.

Trials are admitted per org on the vqueue eval lane, so a task can sit in pending / queued / scheduled for a while before its sandbox exists. The adapter waits through those patiently and the idle clock does not start until the sandbox is running, so queueing does not eat the trial's budget. It does eat a Harbor worker slot, though: Harbor's n_concurrent_trials defaults to 4, and setting it above the org's eval_concurrency just parks the excess trials inside start. Match the two.

Directories, and which ones exist

Harbor's blessed set is EnvironmentPaths, a frozen dataclass — there is no task field for declaring extra directories. Harbor writes into these and start creates the /logs children, but none can be created directly under a root-owned / as uid 1000, so the adapter mounts them as compose named volumes, which the platform renders as emptyDir under the pod's fsGroup: 1000.

Path Mounted What writes it
/logs yes Harbor. Its children are the directories Harbor bind-mounts on a mounted provider (agent/, verifier/, artifacts/, and user-agent/ for a simulated user), and Harbor creates only verifier/ itself, before verifying, so start creates each one inside the mount once the sandbox accepts commands
/logs/verifier/reward.{txt,json} — the verifier's result, read back as the trial's reward
/tests yes the Verifier, after the agent runs
/solution yes the OracleAgent only
/harbor/skills no an agent reads it, guarding on is_dir — an empty mount would answer "yes, skills" where the truth is "no skills"

Anything else a task needs writable — /app, /srv, /data, whatever its own Dockerfile used — is the task's to declare, in the compose document it already sends:

services:
  main:
    volumes: [task-app:/app]
volumes:
  task-app:

main accepts volumes and depends_on and nothing else, and the platform mounts them on the agent container. This needs no new field anywhere.

What a volume does not give you. It is an empty writable directory, not the task image's contents. The Runtime image fills the main slot, so everything the task's own Dockerfile set up — a seeded /app, installed packages, fixture files — is absent, and mounting a volume over such a path would mask whatever the Runtime image had there. This closes "the directory does not exist and is not writable"; it does not close "the task's files are missing". A task that needs its own filesystem wants a Runtime built from its image.

Transfer artifacts

Uploads are staged through the Files API as raw (encryption: false) files stamped purpose: harbor_transfer. The adapter attempts best-effort deletion as soon as the sandbox has the bytes. An evals API key includes files:delete, so direct CLI orchestration can soft-delete their rows; the Operator session deliberately lacks that destructive scope, so its delete is refused and suppressed rather than turning a successful trial into a cleanup failure. No TTL or transfer-file sweep currently exists. The Files API delete does not itself erase blob bytes. Download outputs also use raw storage (metadata.encryption=false) and have the same retention limitation. This applies to Harbor's remaining test-input, prompt, replay, sidecar and result transfers; candidate Git checkout and setup do not create transfer Files.

Trial Task cleanup archives the remote task through the SDK's tasks.archive operation (tasks:write), retaining its record for inspection and platform teardown judging. Harbor's --no-delete skips this call and leaves the remote task available for debugging, subject to its configured wall-clock ceiling. Archive requests asynchronous teardown of the sandbox and sidecars; acceptance does not confirm that resources have stopped. Harbor does not poll for teardown. Cleanup remains best effort: an API error is logged at debug level and suppressed so it does not turn a valid verifier score into completed_with_errors or mask an agent or verifier failure. The task's configured wall-clock ceiling remains the fallback. This does not change platform judge eligibility.

Known gaps, all of them in the command contract rather than this adapter: exec's user cannot be honoured (the sandbox runs every command as its own user, so the argument is ignored and a command that needed another user fails on its own exit code); the network policy is fixed when the pod is created, so set_network_policy mid-trial is refused; and a command row carries a bounded output tail rather than unbounded output, so a chatty command wants redirect-then-download. stop_service is the one per-service operation this adapter cannot offer — a pod's containers stop together — so sidecar evidence is collected with the agent still running, which Harbor treats as a warning.

Sidecar artifact collection and verifier collect hooks do work: service_exec and service_download_file address one compose service through the command's own service field, the directory walks are generic over that pair, and the plane refuses a service the task did not declare in its own compose. A sidecar command inherits none of the agent's workdir, user or persistent environment, as on every other compose-capable provider.

What the runtime adapter does

The trial image is the Runtime image the Data Plane published for the Recipe, so the Recipe (/opt/introspection/recipe), Pi, the Recipes extension, and the Introspection CLI are already inside it; or, on local Docker, a task's own image with the toolchain installed, where the environment prepared the Recipe at the same path. The adapter installs nothing. Per trial it:

  1. checks that the image's Introspection CLI is 0.78.0 or later and fails the trial's agent setup (CliTooOldError) when it is older or missing: an older introspection local would not detect the Runtime sandbox. Outside a Runtime sandbox it also checks that pi is on PATH and lists the Recipes extension (ToolchainMissingError). Both errors name the fix: FROM introspection-runtime-agent:latest, or on a Node 24 image RUN npm i -g @introspection-ai/cli@<version> && introspection setup --yes;

  2. uploads the Harbor instruction, or the CLI-selected replay prompt, to /tmp/introspection-eval-prompt.md and runs the baked Recipe through the image's CLI from the workspace, passing the file's bytes as the message:

    cd /workspace && export INTROSPECTION_WORKSPACE_DIR=/workspace
    introspection local --work-dir /opt/introspection/recipe \
      --runtime <slug> --agent <agent> --yes --non-interactive -- \
      --session-dir /logs/agent/pi/sessions --model <model> \
      --mode json --approve --print "$prompt"
    

    Inside the Runtime image introspection local detects the Runtime sandbox and starts the image's own Pi (or pi on PATH, as in the Operator image) with the Recipes extension, the Runtime's provider extension, the proxy and telemetry preloads and INTROSPECTION_LOG_LEVEL=WARN, so the adapter keeps no launcher of its own. --runtime is left out when the run pins a Runtime version, whose image holds one manifest.

    A production-conversation replay adds --replay-context, which creates the temporary native Pi session immediately before Pi starts; the CLI then restores the agent's declared model itself. So does a task whose environment/history.json holds the turns before its own, as introspection eval add writes it for a later turn of a conversation: the adapter uploads those turns as the replay context, so Pi answers instruction.md with the conversation so far, as a Vitest row's input.history does. A task without the file, or with an empty list, starts Pi with the instruction alone. The native adapter and an ACP trial refuse a task with history. Every step after the first adds --continue when the run resumes (see Multi-step tasks);

  3. captures Pi's JSON event stream as pi.jsonl under the agent log directory and converts it into trajectory.json (ATIF) with token usage, cost, and the observed model, which introspection eval run verifies against the agent YAML.

The adapter itself never uploads or patches Recipe source. An exact-commit evaluation uses the selected Runtime version's published bake when available; its ordinary cold path otherwise materializes that version's requested SHA before this adapter runs.

E2B through scoped Data Plane egress

Operator can use E2B without receiving a real E2B, OpenAI, or Anthropic key. Configure E2B in the Data Plane's sandbox_providers; weight: 0 keeps it out of normal traffic while allowing an explicitly pinned Operator task to use it. Then select the Introspection environment class in the arguments forwarded to Harbor, naming the Runtime's published template:

introspection eval run --runner harbor --path evals/refund-task -- \
  --env introspection_harbor.environment:IntrospectionE2BEnvironment \
  --environment-kwarg "template_name=$(introspection runtimes get "$RUNTIME_ID" \
    --query image_build_metadata.external_image_name -o json | jq -r .)"

Harbor's native E2B implementation still owns preflight, sandbox creation, resources, and networking. The adapter only:

  • skips Harbor's template build when template_name names a published Runtime template, and defaults the task workdir to the baked Recipe;
  • copies the scoped public-egress contract (INTROSPECTION_TOKEN, INTROSPECTION_PUBLIC_EGRESS_URL as the trial's INTROSPECTION_EGRESS_URL, INTROSPECTION_ENDPOINT_HOSTS, and INTROSPECTION_RELAY_TARGET when set) into the trial through Harbor's supported persistent_env input. A template snapshot carries no session state, so this is the only way the baked Pi learns where its egress is. The Data Plane exchanges the locator for credentials at egress; model provider keys never enter the trial;
  • forwards the host variables named in INTROSPECTION_HARBOR_PASSTHROUGH_ENV (comma-separated) for direct, non-egress runs such as a laptop trial with a developer's own provider key.

The platform Operator image sets INTROSPECTION_HARBOR_ENVIRONMENT=introspection_harbor.environment:IntrospectionE2BEnvironment. When the variable is set, introspection eval run supplies that environment to Harbor unless the caller explicitly passes --env or -e. Ordinary developer installs leave it unset, so the same command keeps the local Docker default (IntrospectionDockerEnvironment, above).

When none of the three Introspection egress variables are set, the same class adds no egress configuration and Harbor uses E2B's normal environment and credentials. A partial egress contract is rejected rather than silently mixing direct and routed traffic.

A Pi Recipe as Harbor's user agent

introspection_harbor.pi_user_agent:PiRecipeUserAgent runs a Pi Recipe on the host as Harbor's user agent: the Recipe talks to the target coding agent (Claude Code first) through Harbor's ACP bridge. What the user agent says, when it stops, what it may share and what it records all belong to the Recipe (for example the owner Recipe in introspection-operator-evals). This adapter only runs Pi and serves the Recipe a few environment primitives. It is a plain Harbor agent in the Harbor process, named by user_agent.import_path in a job config:

agents:
  - name: claude-code
user_agent:
  import_path: introspection_harbor.pi_user_agent:PiRecipeUserAgent
  model_name: openai/gpt-6-luna          # Pi's model
  kwargs:
    recipe_dir: recipes/owner            # the Recipe folder on the host
    tools: [say, share_media, done]      # the model-callable tools: Pi's --tools
    agent: owner                         # optional: the Recipe agent
    inputs:                              # optional: JSON data for the Recipe
      task_dir: use-cases/outbound-desk/tasks/create-1
  bridge:
    kind: acp

Other kwargs: prompt (Pi's first message; default: the trial's instruction), target_capabilities (default: what the target's ACP session advertised, read with acpx sessions show), workspace_dir (where relative upload_file targets go; default: the container's working directory), exec_timeout_sec (default 900) and pi_command. The job config's user_agent.env is added to Pi's environment as given (see Pi's environment).

tools is required: it lists the Recipe's own tools the model may call, and goes to Pi as --tools. An entry is a name or a pattern with *. An entry that names or matches codemode, tool_search or an MCP tool (mcp__...), a +name/-name entry, or one with a comma or whitespace (Pi splits --tools on commas, so say,codemode would enable codemode) is refused, since through any of them the model could call the primitives itself (see below).

Its setup is empty: Harbor sets up the user agent before the target and the bridge, so acpx is only there once run starts. run starts pi --print --mode json --no-context-files --recipe <recipe_dir> --tools <tools> once, for the whole conversation, in a fresh host temp directory. Pi's exit ends the run, and a non-zero exit fails it.

Harbor 0.24 calls only the user agent's populate_context_post_run in such a trial, so the target would write no /logs/agent/trajectory.json and a verifier would read an empty conversation (harbor#3971). The user agent's hook therefore also runs the target's own hook on the target's logs (Claude Code's session JSONL, a Recipe's GenAI spans), with the target made again from the trial's config.json. The target's trajectory covers every prompt the owner sent; its usage is kept under metadata.target in the trial's agent result.

The user agent's trajectory

The user agent writes its own ATIF, as the target does: Pi gets PI_RECIPE_GENAI_SPANS_FILE in its host run directory, and run projects those spans to <trial>/user-agent/trajectory.json right after the run's directory is copied there. That happens on every exit of run, including Pi failing and Harbor's timeout, so a subclass can read the file in its own finally. A tool result's structured details is at observation.results[].extra.details (introspection-sdk 0.35.0 or newer).

User agent Target (Recipe adapter)
Spans Host run directory, copied to <trial>/user-agent/ /logs/agent/genai-spans.json
trajectory.json written by run, on any exit populate_context_post_run
Verifier reads /logs/user-agent/trajectory.json /logs/agent/trajectory.json
Pi never started No file, no error No file, no error
Pi reached the model, spans missing or unreadable Harbor records the error, the target's hook and the verifier still run The trial fails with the same error
Usage The trial's agent_result metadata.target, when a user agent runs the trial

The file reaches a verifier on either kind of environment. Docker bind-mounts the directory, and for a non-mounted one Harbor 0.24 uploads <trial>/user-agent/ to /logs/user-agent/ after the post-run hook and before a shared verifier. A verifier with environment_mode = "separate" gets neither role's logs, only the task's artifacts. Harbor refuses a user agent on a multi-step task, so a trial has one user-agent conversation.

agent_result, and with it the tokens and cost in the introspection eval run report, is the user agent's alone in such a trial. The target's are in metadata.target, and are in no report total. The report's Model line is still the target's: it reads only the agent/ trajectory.

Pi's environment

Pi gets an explicit environment, not the Harbor process's. Harbor can run in an Operator Runtime, where passing everything would hand the Recipe the Operator's INTROSPECTION_TOKEN, and the Runtime's telemetry preload would report Pi's calls into the Operator's conversation. Pi keeps:

  • the process basics: PATH, HOME, USER, SHELL, TMPDIR, LANG, LC_*, TERM, XDG_*, NVM_* and NODE_OPTIONS and its neighbours;
  • the network path: the *_PROXY variables and the CA bundle variables;
  • provider credentials: any *_API_KEY except E2B_* and INTROSPECTION_*, ANTHROPIC_*, OPENAI_*, AWS_*, AZURE_*, GOOGLE_*, GCLOUD_PROJECT, HF_TOKEN and COPILOT_GITHUB_TOKEN;
  • Pi's own: PI_* and MCPORTER_*;
  • a Runtime's egress contract: INTROSPECTION_EGRESS_*, INTROSPECTION_ENDPOINT_HOSTS, INTROSPECTION_RELAY_TARGET, INTROSPECTION_PROXY_KEY and INTROSPECTION_LOG_LEVEL.

A Recipe that reads any other variable names it in the job config, which is also how to hand Pi something on purpose:

user_agent:
  import_path: introspection_harbor.pi_user_agent:PiRecipeUserAgent
  env:
    RECIPE_SETTING: ${RECIPE_SETTING}

Model access takes the target's managed path. With INTROSPECTION_TOKEN and INTROSPECTION_PUBLIC_EGRESS_URL in the Harbor process and a provider/model model_name, Pi presents the session locator as that provider's key, and the egress proxy swaps it for the real credential, so a Runtime with no provider keys can run it. The locator is the token's value under the provider's name: the Recipe can still read it, but the telemetry preload, which looks for INTROSPECTION_TOKEN, stays off. On a host with the Runtime's provider extension Pi loads that too. Without managed egress, Pi uses the host's provider keys or its own ~/.pi login, as before.

The primitives

The primitives are MCP tools on a streamable-HTTP server inside the Harbor process, on 127.0.0.1 with a per-run bearer token. Pi gets both in its environment:

Variable Value
INTROSPECTION_USER_AGENT_URL The endpoint, http://127.0.0.1:<port>/mcp
INTROSPECTION_USER_AGENT_TOKEN Send it as Authorization: Bearer <token>

A Recipe extension registers the server with Pi's own MCP client and calls a primitive from its tools with ctx.executeTool:

export default function (pi) {
  pi.registerMcpServer("harbor", {
    url: process.env.INTROSPECTION_USER_AGENT_URL, // Pi refuses `${...}` in `url`
    headers: { Authorization: "Bearer ${INTROSPECTION_USER_AGENT_TOKEN}" },
    exposure: "codemode",
  });
  pi.registerTool({
    name: "say",
    // label, description, parameters ...
    async execute(id, params, signal, onUpdate, ctx) {
      const outcome = await ctx.executeTool(
        "mcp__harbor__prompt_target",
        { prompt: [{ type: "text", text: params.text }] },
        { signal, onUpdate },
      );
      const { reply, exit_code } = JSON.parse(outcome.result.content[0].text);
      // ... the Recipe's own rules and log ...
    },
  });
}

Pi expands ${NAME} in headers, so the token stays out of the extension's code; url must already be a URL when it is registered, so it is read from process.env. tests/fixtures/synthetic-recipe is a complete Recipe in this pattern.

exposure: "codemode" keeps the primitives out of the model's tool list, but Pi turns its codemode tool on as soon as such a server connects, and a codemode script can call them. That is why tools is a --tools allowlist without codemode: the model reaches the target only through the Recipe's own tools, and the Recipe's limits (a message cap, media rules) can't be bypassed. executeTool doesn't depend on the model's tools, so the extension still reaches every primitive.

A prompt can wait minutes for the target's turn, longer than Pi's 60-second MCP request timeout. Pi sends a progressToken with each call, and the server answers with a progress notification every 15 s while the call runs. Each one resets the timeout and reaches the extension's onUpdate. Any other MCP client works too: one JSON-RPC request per POST, answered as a server-sent event stream, with a comment every 15 s when the request has no progressToken.

A result is the tool's JSON object, in structuredContent and as the text content (through executeTool, outcome.result.content[0].text). isError: true, with the message as text, means the primitive could not run: bad arguments, or the environment raised. A non-zero exit of acpx or of a command is data, in exit_code.

Tool Arguments Result
prompt_target prompt: ACP's own prompt array of content blocks (ContentBlock[]), sent as-is. Each block needs a type; at least one block is needed. {reply, exit_code}. One ACP prompt through acpx prompt --file; the blocks are written as JSON on the host (kept as prompts/NNN.json), uploaded and sent. Prompts run one at a time. acpx prints an error on stdout too, so only exit_code tells a reply from an error.
upload_file source: a host path; target: a container path, relative to the workspace or absolute {path}, the container path written. Parent directories are created.
exec command: a shell command; cwd?; timeout_sec? (default exec_timeout_sec) {exit_code, stdout, stderr}
context none {logs_dir, target_capabilities, workspace_dir, instruction, inputs}: see below

context fields:

  • logs_dir: the run's host directory. Whatever the Recipe writes there ends up in <trial>/user-agent/.
  • target_capabilities: the target's ACP prompt capabilities, for the Recipe's own gating. acpx refuses an audio or resource block the target didn't advertise.
  • workspace_dir: the container directory relative upload_file targets resolve against.
  • instruction: the trial's instruction.
  • inputs: the job's inputs, as given.

A call lasts no longer than its connection. When Pi exits or is killed, or Harbor cancels the trial, every call still running is cancelled and awaited before anything is copied.

What the target can see

Harbor bind-mounts the trial's log directory into a Docker container, so the target could read <trial>/user-agent/ while it runs. So logs_dir in context is a host temp directory, not that one. The temp directory (Pi's pi.jsonl and pi-stderr.log, sessions/, prompts/, and what the Recipe wrote) is copied into <trial>/user-agent/ only once Pi has exited and every primitive call has ended. The token is never written to a file. Only what the Recipe sends with prompt_target, upload_file or exec reaches the container. Claude Code's ACP shell tool (mcp__acp__Bash) does see the target's credential: in a trial, CLAUDE_CODE_OAUTH_TOKEN was set in its environment, so a command the agent runs there has it.

The trial result's agent metadata carries user_agent: {exit_code, prompts}, plus metadata with the contents of metadata.json, when the Recipe wrote one to logs_dir.

A Recipe as the target

The runtime adapter is an ACP target too, so the user agent can talk to a Recipe instead of Claude Code:

introspection eval run --chain use-cases/<case>/chain.yaml --agent <recipe agent> \
  --user-agent introspection_harbor.pi_user_agent:PiRecipeUserAgent --bridge acp ...

Harbor's bridge then calls the adapter's acp_install after install: it installs pi-acp@0.0.34 into /tmp/introspection-acp and writes two launchers there. pi runs Pi with the same Recipe, agent, model and skills as an authored run, through introspection local (in the Runtime image with its provider extension and preloads, in a task's own image with the Pi introspection setup installed); pi-acp adds --mode rpc and the session itself. acp_command starts pi-acp with that launcher as PI_ACP_PI_COMMAND, PI_ACP_ENABLE_EMBEDDED_CONTEXT=true (so the Recipe may send resource blocks), Pi's sessions under /logs/agent/pi/sessions and the spans in /logs/agent/genai-spans.json. Pi runs in the session's directory, the task's workspace, and its stderr goes to /logs/agent/pi-stderr.log. Under managed egress acp_env hands the provider key on to acpx, pi-acp and Pi, as Harbor does with the target's --ae variables; nothing is written into the container. A trial that is not driven over ACP installs nothing more. The native adapter is no ACP target: the platform runs its Recipe, so there is no process to attach to.

Where this is heading

The adapter has two seams: how the Recipe is launched (run) and how the result is collected (populate_context_post_run). Today the launch executes Pi inside a Harbor-owned sandbox and the result is read back from that sandbox as pi.jsonl. A bare pi process exports no telemetry, so nothing from a trial reaches the platform's conversation store.

The intended end state keeps Harbor as the trial orchestrator but moves both seams onto the platform: the launch becomes a task on the Runtime, and the result is read from that task's conversation and judgement events, which the Runtime already exports. That removes the direct E2B dependency and the file-based verifier from the path, and a Recipe judge becomes the reward.

End-to-end MCP example

recipe-harbor-mcp-agent runs a Harbor trial in E2B against a published Runtime whose Recipe calls an authenticated MCP server. See that repository's README for the setup and the scripts/run-e2b.sh helper.

Explicit offline judgement subjects

A verifier that calls introspection judges evaluate can publish its completed judgement through the standard OTLP judgement event by passing both --runtime-slug <slug> and --conversation-id <evaluated-conversation-id>. Use the evaluated trial's conversation ID, not the parent command's ambient conversation. The YAML name supplies the judge slug; each grading attempt mints a UUIDv7 result ID. The existing session token and collector must be available to the verifier. Production judges run keeps its registry identity and deterministic deduplication IDs.

Metadata

Release files for introspection-harbor 0.18.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for introspection-harbor 0.18.1
File Size Uploaded Provenance Actions
introspection_harbor-0.18.1.tar.gz 370.2 kB With provenance Details for introspection_harbor-0.18.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for introspection-harbor 0.18.1
File Interpreter ABI Platform Actions
introspection_harbor-0.18.1-py3-none-any.whl Python 3 none any Details for introspection_harbor-0.18.1-py3-none-any.whl

Total release size: 484.0 kB

Release files / introspection_harbor-0.18.1.tar.gz

Download URL introspection_harbor-0.18.1.tar.gz
Size 370.2 kB
Tags Source
SHA-256 checksum
How to use checksums
6644d987fd34d2f93b7e21bfb7c14a7170e4a2aef364b5dc037853c1cd789d12
BLAKE2b-256 checksum
How to use checksums
219db90a173e14b2f87f46b08204647ac008ff9d4da267e5462c3004f59adda4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / introspection_harbor-0.18.1-py3-none-any.whl

Download URL introspection_harbor-0.18.1-py3-none-any.whl
Size 113.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
afc799be3c7c861689ac7d45a2be5b6c3209d99bedd8330e12e71862fb01abd8
BLAKE2b-256 checksum
How to use checksums
015d9a04bc9c7b0c8793cac19ca3d3992dbd64dc583df0e52d5d4f5ac5a2caa0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Security for introspection-harbor 0.18.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for introspection_harbor-0.18.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.18.1 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page