AfterAI Runner

Run AI agent evaluations and emit AIS signals — on your own infrastructure.

• Eval: poll AfterAI for pending eval jobs, run prompts against your agent, submit responses for scoring
• AIS emission: send pre-aggregated metric signals to the AfterAI API from local artifact files
• No inference proxy: the runner runs on your infra; AfterAI never sees your agent's API keys or endpoints

Install

pip install afterai-runner

Or from source:

cd afterai-runner && pip install -e .

Config

Eval

Run evaluations against your agent. AfterAI generates a synthetic dataset, you approve it in the app, then the runner executes it locally and submits results for scoring.

Quick start

# 1. Create a starter agent config
afterai-runner eval-init

# 2. Edit afterai-agent.yaml with your agent details (see below)

# 3. Run — polls for pending eval jobs and processes them
afterai-runner eval

Agent config (afterai-agent.yaml)

Azure AI Foundry (v2 — recommended)

agent:
  type: foundry
  foundry_version: v2
  project_endpoint: https://YOUR_RESOURCE.services.ai.azure.com/api/projects/YOUR_PROJECT
  agent_name: your-agent-name
  # Auth: set AZURE_FOUNDRY_TOKEN env var, or run `az login`

Azure AI Foundry (v1 — legacy)

agent:
  type: foundry
  foundry_version: v1
  project_endpoint: https://YOUR_PROJECT.services.ai.azure.com
  agent_id: asst_YOUR_AGENT_ID
  api_key: ${AZURE_API_KEY}

Generic HTTP

agent:
  type: http
  url: https://your-agent.example.com/chat
  headers:
    Authorization: Bearer ${YOUR_API_KEY}
  body_template: '{"message": "{prompt}"}'
  response_path: response.text

Add afterai-agent.yaml to .gitignore — keep your credentials local.

Eval commands

# Poll continuously (default: every 10s)
afterai-runner eval

# Process pending jobs once and exit
afterai-runner eval --once

# Custom poll interval
afterai-runner eval --poll-interval 30

# Custom config path
afterai-runner eval --agent-config /path/to/my-agent.yaml

Generate a dataset from your Foundry agent (optional)

If your agent is on Foundry, the runner can fetch its definition and generate an eval dataset automatically:

afterai-runner dataset-generate

This calls the AfterAI API to generate a synthetic dataset based on your agent's instructions and tools. You then review and approve it in the app before any eval runs.

AIS Emission

Send pre-aggregated metric signals from local artifact files to the AfterAI API.

# Run ingestion from a manifest
afterai-runner run --manifest manifest.yaml

# Print payloads without sending
afterai-runner run --manifest manifest.yaml --dry-run

# ACE-from-AIS rule: read AIS signals, emit inferred ACE if threshold met
afterai-runner ace-from-ais --min-signals 2 --hours 24
afterai-runner ace-from-ais --min-signals 2 --severity high,critical --dry-run

Manifest example

version: "2026-02"
artifacts:
  - system: my-system-01
    path: /path/to/D1-2026-02-03.json
  - system: my-system-01
    path: /path/to/W1-2026-W05.json

Artifact example

One file = one signal. D1-2026-02-03.json:

{
  "signal_key": "D1",
  "cadence": "daily",
  "window": {
    "start": "2026-02-03T00:00:00Z",
    "end": "2026-02-03T23:59:59Z"
  },
  "metrics": {
    "score_mean": 0.87,
    "fail_rate": 0.03
  }
}

License

MIT