awaf-cli

The reference implementation of the AWAF open specification. Catch agent architecture regressions before they ship.

Scores across 10 architectural pillars defined by the AWAF open specification. Designed to run periodically -- nightly, weekly, or on-demand before releases -- not on every commit. Each run makes 10 LLM calls; run it when architecture decisions change, not when typos are fixed.

No dashboards that need a legend. No compliance jargon. One number per pillar, one finding per issue, one fix per finding.

Install

pip install awaf

Requires Python 3.11+. Bring your own model and API key.

Provider Support

awaf-cli is model-agnostic. Use any supported LLM provider — no vendor lock-in.

Default provider: anthropic with claude-haiku-4-5-20251001. Scores are calibrated on Claude; other providers may yield slight variance.

API Keys from .env

awaf automatically loads a .env file in the current directory at startup. Keys already set in the environment take precedence.

Create a .env file next to your project:

ANTHROPIC_API_KEY=sk-ant-...
# OPENAI_API_KEY=sk-...
# GOOGLE_API_KEY=...

Then run normally — no export needed:

awaf run
awaf run --pillar foundation

If you prefer to load .env manually before running:

# bash / zsh
export $(grep -v '^#' .env | xargs) && awaf run

# PowerShell
Get-Content .env | ForEach-Object { $k,$v = $_ -split '=',2; [System.Environment]::SetEnvironmentVariable($k,$v) }; awaf run

Quickstart

# Default: Anthropic (.env or export)
export ANTHROPIC_API_KEY=sk-ant-...
awaf run

# OpenAI
export OPENAI_API_KEY=sk-...
awaf run --provider openai --model gpt-4o

# Azure / GitHub Copilot
export AZURE_OPENAI_API_KEY=...
awaf run --provider azure --model gpt-4o --azure-endpoint https://your-resource.openai.azure.com --azure-deployment gpt-4o

# LiteLLM (Bedrock, Groq, Ollama, etc.)
awaf run --provider litellm --model bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0

   _      _  _  _    _      ___
  /_\    | || || |  /_\    | __|
 / _ \   | \/ \/ | / _ \   | _|
/_/ \_\   \_/\_/  /_/ \_\  |_       Agent Well-Architected Framework

  PREFLIGHT
  Artifacts          12,450 tokens  (12 files)
  Context window    128,000 tokens  (gpt-4o)
  Per-pillar est     13,350 tokens  (10% of window)
  Total est         133,500 tokens  (10 pillars × ~13,350)
  Cost est               ~$0.0093
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

AWAF Assessment: my-agent
AWAF v1.4  |  2026-03-15  |  openai / gpt-4o
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Overall Score    78/100   Near Ready
  Close to production. Address findings before deploying.

  Scale: Production Ready >=85 · Near Ready >=70 · Needs Work >=50
         High Risk >=25 · Not Ready <25
  Foundation <40 = automatic FAIL regardless of overall score.
  Tier 2 pillars (Reasoning, Controllability, Context Integrity) carry 1.5x weight.

┌──────────────────────┬───────┬──────────────┬────────────┬─────────┐
│ Pillar               │ Score │ Progress     │ Confidence │  Status │
╞══════════════════════╪═══════╪══════════════╪════════════╪═════════╡
│ TIER 0 -- FOUNDATION                                               │
├──────────────────────┼───────┼──────────────┼────────────┼─────────┤
│ Foundation           │    85 │ [########  ] │ verified   │    PASS │
╞══════════════════════╪═══════╪══════════════╪════════════╪═════════╡
│ TIER 1 -- CLOUD WAF ADAPTED                                        │
├──────────────────────┼───────┼──────────────┼────────────┼─────────┤
│ Op. Excellence       │    74 │ [#######   ] │ verified   │         │
│ Security             │    82 │ [########  ] │ verified   │         │
│ Reliability          │    71 │ [#######   ] │ verified   │         │
│ Performance          │    80 │ [########  ] │ verified   │         │
│ Cost Optim.          │    65 │ [######    ] │ partial    │         │
│ Sustainability       │    79 │ [########  ] │ verified   │         │
╞══════════════════════╪═══════╪══════════════╪════════════╪═════════╡
│ TIER 2 -- AGENT-NATIVE  (1.5x weight)                              │
├──────────────────────┼───────┼──────────────┼────────────┼─────────┤
│ Reasoning Integ.     │    71 │ [#######   ] │ partial    │    1.5x │
│ Controllability      │    78 │ [########  ] │ verified   │    1.5x │
│ Context Integrity    │    80 │ [########  ] │ verified   │    1.5x │
└──────────────────────┴───────┴──────────────┴────────────┴─────────┘

  FILES ANALYZED     12 files
  TOKENS             133,450 in / 41,000 out  (peak call: 11% of 128K window)
  COST (est)         ~$0.0093
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  FINDINGS  (ordered by severity)
  [High     ]  Cost Optim.         No session budget cap; runaway token spend possible
  [Medium   ]  Reasoning Integ.    Evals present but hallucination rate not measured
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  RECOMMENDATIONS
  Cost Optim.         Add AWAF_SESSION_BUDGET_USD env var and wire hard stop in
                      agent loop before tool dispatch
  Reasoning Integ.    Instrument LangSmith eval run to capture hallucination rate
                      alongside tool selection accuracy
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  TO IMPROVE THIS ASSESSMENT
  Share LangSmith or Braintrust eval output to upgrade Reasoning Integ.
  from partial to verified
  Share token usage dashboard or budget alert config to verify Cost Optim.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Per-Project Config

# awaf.toml
[project]
name = "my-agent"

[provider]
name = "openai"              # anthropic | openai | azure | google | litellm
model = "gpt-4o"
api_key_env = "OPENAI_API_KEY"   # defaults to provider standard env var

# Azure / Copilot specific
# name = "azure"
# model = "gpt-4o"
# api_key_env = "AZURE_OPENAI_API_KEY"
# azure_endpoint = "https://your-resource.openai.azure.com"
# azure_deployment = "gpt-4o"
# azure_api_version = "2025-01-01-preview"

# LiteLLM — any model string LiteLLM supports
# name = "litellm"
# model = "bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0"

[thresholds]
overall_fail = 60      # fail if the overall score < 60
tier2_fail = 50        # fail if the Tier 2 (Agent-Native) average < 50
warn_only = false      # true = report breaches but never exit non-zero
# Band-drop regression is automatic and needs no config: a run that falls into a
# lower readiness band than the previous run for the same project fails the build.

[files]
# paths: controls what gets ingested. --paths CLI flag overrides this.
paths = ["agents", "awaf", "pipeline.py", "main.py", "models.py", "utils"]
# agent_patterns: used ONLY for CI change detection (exit 3 when nothing relevant changed).
agent_patterns = ["agents/**/*.py", "tools/**/*.py", "pipelines/**"]
exclude = ["tests/**", "docs/**"]

[ci]
enabled = true
schedule = "0 9 * * 1"        # cron (UTC): only run when this schedule fires
change_detection = true        # skip if no files changed under watched paths
# watch_paths is optional — omit it and CI uses [files] paths above
watch_paths = [
    "src/agents",
    "src/signals",
]

[reporting]
post_pr_comment = true
terminal_format = "compact"    # compact | full | json

CI Config Fields

Agent-Architecture Graph (token-efficient evidence)

By default, awaf precomputes a compact, pillar-shaped agent-architecture graph of your repository. Instead of sending the raw code dump to each pillar evaluator, awaf extracts a graph of agents, tools, guardrails, data stores, and context sources plus a file-role manifest, then sends that graph (plus targeted code slices for each pillar) as evidence. This replaces the raw-file path.

Cost Savings

Replacing the raw-code dump with the graph cuts per-run input tokens by roughly 40-50%. Because the graph is cached by a content hash of the repository, an unchanged re-run reuses it and skips extraction entirely, so repeat runs approach free.

Quality and Reliability

No more silent truncation on large repositories (the extractor sees the whole repo, not just the first 40k tokens). Findings gain validated file:line anchors (validated against the analyzed view of each file, which awaf minifies, so line numbers are close but not always exactly the raw-source line). Each pillar receives focused evidence instead of the full dump. The graph is on by default and fully optional: any extraction or cache failure automatically falls back to the raw-dump path, so a run is never worse than before.

Usage

The graph is enabled by default. To disable it for a single run:

awaf run --no-graph

To force a rebuild of the cached graph:

awaf run --refresh-graph

To inspect what was extracted:

awaf graph                 # view the graph summary
awaf graph --json          # raw JSON
awaf graph --refresh       # rebuild before inspecting

Configuration

Configure the graph via the [graph] table in awaf.toml:

[graph]
enabled = true
extract_tokens = 150000    # one-time budget for the extraction call
slice_budget = 12000       # per-pillar cited-code budget
context_lines = 20         # lines around each anchored node
cache_max = 8              # graphs kept in the on-disk cache
starvation_retry = true    # re-feed raw slices to a low-confidence pillar once

Environment variable overrides (take precedence over awaf.toml):

AWAF_GRAPH=0                          # disable (0 or 1)
AWAF_GRAPH_EXTRACT_TOKENS=150000
AWAF_GRAPH_SLICE_BUDGET=12000
AWAF_GRAPH_CACHE_MAX=8

The graph cache lives next to awaf.db in a graph_cache/ directory.

What changes per run

The shape of the change (actual numbers depend on your repo size, model, and provider):

• Raw path: the full minified dump (up to the AWAF_MAX_ARTIFACTS_TOKENS budget) is the evidence for every pillar. Prompt caching on Anthropic makes the within-run re-reads cheap, but every fresh run re-sends the whole dump.
• Graph path: one extraction pass builds the graph, then each pillar sees a small shared graph block plus only its cited code slices. On an unchanged re-run the cached graph is reused and extraction is skipped, so the per-run input shrinks to the graph block plus slices.

Run awaf graph to see the actual node, edge, and file-role counts, the token cost of extraction, and the cache status for your own repository.

CI Integration

What CI gates are valid — and which ones are not

LLM assessments are non-deterministic even at temperature=0.0. Repeated runs on identical artifacts typically agree within ±3–5 points, but a single run is not a reliable point estimate. This has direct consequences for how you configure thresholds:

Before setting overall_fail, establish a baseline:

# Run 5–10 times on the same codebase, then inspect variance
for i in {1..5}; do awaf run; done
awaf history

# mean=88, σ=2  → threshold of 85 is a reliable gate
# mean=88, σ=12 → no absolute threshold is reliable; rely on the automatic band-drop gate only

If σ > 5, switch to a stronger model (--model claude-sonnet-4-6) — it shows significantly less variance than Haiku — or drop the absolute threshold and use regression detection only.

To measure mean and σ across several models in one pass (for example, over the bundled examples/good-agent and examples/bad-agent), run the calibration harness scripts/calibrate.py. See CALIBRATION.md for how to run it and interpret the numbers.

GitHub Actions

The AWAF Assessment action installs the CLI and runs awaf run --ci. Gates (overall_fail, tier2_fail, warn_only) live in a committed awaf.toml; the job fails on a threshold breach or a band drop, and is skipped (exit 3) when no agent-relevant files changed.

name: AWAF Assessment
# Architecture changes slowly and each run makes ~10 LLM calls, so a weekly
# schedule or on-demand run is usually the right cadence, not every commit.
on:
  schedule:
    - cron: '0 6 * * 1'   # every Monday at 06:00 UTC
  workflow_dispatch:         # on-demand: run before releases or after major changes

jobs:
  awaf:
    runs-on: ubuntu-latest
    env:
      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}   # the CLI reads the key from env
    steps:
      - uses: actions/checkout@v4
      - uses: YogirajA/awaf-action@v1
        with:
          provider: anthropic                    # anthropic | openai | azure | google | litellm
          # model: claude-haiku-4-5-20251001     # optional; omit to use the provider default

You can equivalently pass the key as an action input (anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}); an empty input never overwrites a key already in the environment.

Prefer to run the CLI directly (custom runners, or without the action)? Install awaf and run awaf run --ci yourself, exactly as in the GitLab example below.

Recommended cadence: weekly schedule or on-demand before releases. Running on every PR makes sense only for teams actively refactoring agent architecture. For most teams, weekly is sufficient -- architecture changes slowly.

If you do run on PRs, use on: pull_request with a paths: filter so only agent-relevant changes trigger it. AWAF also exits 3 automatically when no relevant files changed (controlled by [ci] watch_paths, falling back to [files] paths, then agent_patterns).

GitLab CI

awaf:
  image: python:3.12-slim
  script:
    - pip install awaf
    - awaf run --ci --paths .
  variables:
    AWAF_PROVIDER: anthropic
    # ANTHROPIC_API_KEY is set as a masked CI/CD variable in project settings.
    # Gates (overall_fail, tier2_fail, warn_only) live in a committed awaf.toml.

Skill Eval Grading (awaf eval-skill)

The awaf-skill Claude Code skill ships with five eval cases (skills/awaf/evals/evals.json), each a prompt plus an expected report summary and a list of natural-language expectations about the resulting report. awaf eval-skill runs those cases end to end and grades the output:

uv run awaf eval-skill --skill-dir ../awaf-skill --output eval-metrics.json

For each case it runs the skill prompt through the configured provider, applies deterministic report-shape and band-consistency checks, then has a judge model (--judge-model, which defaults to the subject model; the nightly workflow uses claude-opus-4-5) score every expectation against the report. It writes a metrics JSON (--output, default eval-metrics.json) and exits non-zero if the pass rate falls below the gate (--gate, default 0.85) or any deterministic check fails.

This corresponds to two CI layers:

• Layer 1 (report shape, band consistency) is pure Python, no LLM calls, and runs in the normal unit-test suite (ci.yml) on every push and pull request, at no API cost.
• Layer 2 (LLM-judge grading) calls a real provider and judge model, so it runs nightly (and on demand) via .github/workflows/eval-grader.yml. That workflow checks out both awaf-cli and awaf-skill, runs eval-skill with ANTHROPIC_API_KEY from repo secrets, and uploads the metrics JSON as a build artifact even when the gate fails.

--provider and --model override the subject provider and model that run the skill, resolved the same way as awaf run (CLI flag, then environment, then awaf.toml).

Exit Codes

CLI Reference

awaf run                                         # assess current directory
awaf run --paths agents/ tools/                  # specific paths
awaf run --ci                                    # CI mode with git context
awaf run --pillar foundation                     # single pillar only
awaf run --provider openai --model gpt-4o        # override provider
awaf run --provider litellm --model ollama/llama3 # local model via LiteLLM
awaf run --parallel                              # concurrent mode (faster, higher cost)
awaf run --delay 10                              # sequential with 10s pause between pillars
awaf run --model claude-opus-4-5                 # override model (default: claude-haiku-4-5-20251001)
awaf history                                     # score history for current project
awaf compare <id1> <id2>                         # diff two assessments
awaf report --format json                        # JSON output for CI artifact upload
awaf report --coverage                           # show files analyzed and skipped
awaf providers                                   # list configured providers and status
awaf eval-skill                                  # grade the awaf skill's eval cases (Layer 2)

Progress is printed as each pillar starts (▸ Evaluating Foundation...). No color codes when stdout is not a TTY. No spinners in CI mode.

Running pillars one at a time

Useful on free-tier API plans or when debugging a specific pillar. Each run saves to awaf.db and contributes to score history.

awaf run --pillar foundation
awaf run --pillar security
awaf run --pillar controllability
# ... pick the pillars you care about

To add a pause between sequential pillar calls (useful on rate-limited API plans):

awaf run --delay 15

What Gets Scored

awaf-cli implements AWAF v1.4 across 10 pillars in 3 tiers. Full pillar definitions and scoring questions are in the specification repo.

Tier 0: Foundation. Can this agent run independently?

Tier 1: Cloud WAF Adapted (1.0x weight). Operational Excellence, Security, Reliability, Performance Efficiency, Cost Optimization, Sustainability

Tier 2: Agent-Native (1.5x weight). Reasoning Integrity, Controllability, Context Integrity

The agent-native pillars are what make AWAF distinct. Cloud infrastructure has no equivalent for them; they exist because agents are not servers. See aradhye.com for the original thinking behind this.

What It Analyzes

awaf-cli reads what is in your repository: Python, TypeScript, Go, YAML, JSON, TOML, Markdown, and PDF files.

It can verify: trust tier enforcement in code, kill switch and cancel implementations, loop detection and budget guards, eval framework presence, sanitization at input boundaries, slice boundary documentation.

It cannot verify (flagged as partial confidence): cloud resource configs not in the repo, whether SLOs are being met in production, runtime hallucination rates, whether circuit breakers are actually firing.

When something cannot be verified, the output says so explicitly. Partial confidence with clear coverage gaps is more useful than a confident score built on assumptions.

Scoring and Bands

AWAF scores are interpreted as bands, not point estimates. LLM-based assessment has inherent run-to-run variance of ±3–10 points; moving within a band is noise. Only crossing a band boundary is meaningful.

Thrashing between bands is expected and should be ignored. A score that moves from 91 to 85 between runs has not regressed — both are Production Ready.

A band change is only meaningful when both conditions are met:

1. Significant agentic code changes are detected (not just docs or tests)
2. Multiple runs confirm the new band

When awaf run detects a band drop, the warning message says:

Band dropped from 'Production Ready' to 'Near Ready' (88 → 72). Confirm with multiple runs (--runs N) before treating as a regression.

To confirm a band change is real:

awaf run --runs 3 --force   # run 3 times, check if all 3 land in the same band

Point scores within a band are not comparable run-to-run. Compare bands, not points.

Score History

Every assessment is stored locally in awaf.db. Score history is tracked per project, per branch, per commit, and per provider/model.

awaf history

my-agent  last 5 assessments
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  2026-02-27  a3f9c12  PR #47   72  -6   openai/gpt-4o       Controllability regression
  2026-02-24  8bc1a33  main     78  +3   anthropic/claude-opus-4-5  Context Integrity improved
  2026-02-21  4de92f1  main     75  +0   anthropic/claude-opus-4-5
  2026-02-18  2ab77c4  main     75  +8   openai/gpt-4o       Security and Reliability up
  2026-02-12  9ff3e21  main     67  —    anthropic/claude-opus-4-5

Six months of CI runs become your architectural changelog.

How It Works

awaf-cli sends your architecture artifacts to the LLM provider of your choice. Each of the 10 AWAF pillars is evaluated by a separate model call running sequentially by default. On Anthropic, the artifact content is sent as a shared cached system block — Foundation writes it to cache, and the remaining 9 pillars read from the same cache key at ~10% of the write cost (~65–80% total cost reduction vs. uncached). Use --parallel for concurrent execution. Results are written to a local SQLite database. No central coordinator. No shared state between pillar evaluations.

Artifacts → Ingestor → Preflight check → [10 Pillar Agents sequentially] → Validator → SQLite → Terminal
                                                    ↑                           ↑
                                        Provider Abstraction Layer       Dead letter quarantine
                                   (Anthropic | OpenAI | Azure | Google | LiteLLM)

The preflight step estimates token usage and cost before any API calls are made, and aborts if the artifact set would overflow the model's context window or exceed a session budget. The validator checks each pillar result for signs of truncation, known pathological scores, or score clustering, and excludes suspect results from the overall score.

The tool is built to be AWAF-compliant itself: choreography over orchestration, vertical slice per pillar, blast radius bounded. See ARCHITECTURE.md.

Environment Variables

# Provider selection (can also be set in awaf.toml)
AWAF_PROVIDER=anthropic          # anthropic | openai | azure | google | litellm
AWAF_MODEL=claude-haiku-4-5-20251001  # optional model override

# API keys — use whichever provider you're running
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
AZURE_OPENAI_API_KEY=...
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com
AZURE_OPENAI_DEPLOYMENT=gpt-4o
GOOGLE_API_KEY=...

# Session controls
AWAF_DB_URL=sqlite:///./awaf.db
AWAF_MAX_ARTIFACTS_TOKENS=40000
AWAF_SESSION_BUDGET_USD=1.00     # approximate; abort before run if preflight estimate exceeds this
AWAF_MAX_CONTEXT_PCT=85          # abort if per-pillar token estimate exceeds this % of context window
AWAF_CONCURRENCY=1               # pillar workers (default 1 = sequential/economical; set higher for --parallel override)
AWAF_LOG_LEVEL=INFO

# Anthropic: prompt caching is enabled automatically. The artifact content is
# placed first in the system prompt as a shared cached block (cache key = artifact
# only). Pillar criteria are appended uncached. Foundation writes the cache;
# pillars 2-10 read from the same key at ~10% of the write cost.
# Cache TTL is ~5 minutes; --runs N within that window gets cache reads on all runs after the first.

# Token counting (Anthropic and Google)
AWAF_EXACT_TOKEN_COUNT=1    # use provider API for exact token counts during ingest
                             # (1 API call per file; default: fast heuristic len(text)//4)

Deployment Modes

The local mode is fully functional with no account required. Cloud and on-prem add team dashboards, cross-project score history, and industry benchmarks. On-prem: no artifacts leave your network. All model API calls use your own API key. No telemetry unless opted in.

Score Badge

[![AWAF Score](https://img.shields.io/badge/AWAF%20Score-94%20Production%20Ready-16A34A?style=flat-square)](https://github.com/YogirajA/AWAF)

Update the score and label after each awaf run to reflect your latest result.

Troubleshooting

Pillars score 0 / "unparseable JSON" warning

Symptom

Pillar 'Foundation' returned unparseable JSON: Expecting ',' delimiter: line 111 column 6 (char 16195)

The pillar gets a score of 0 and confidence: self_reported instead of a real evaluation.

Cause

Some models — particularly smaller or quantized variants — produce JSON that violates the spec when the response is long (code snippets, multi-line findings). awaf-cli attempts automatic repair via json-repair, but repair can fail when the output is severely malformed.

Workaround: upgrade to a more capable model

# Anthropic — Sonnet or Opus handles long structured output reliably
awaf run --model claude-sonnet-4-5
awaf run --model claude-opus-4-5

# OpenAI
awaf run --provider openai --model gpt-4o

# Local via LiteLLM — try a larger quant
awaf run --provider litellm --model ollama/llama3:70b

Or set it permanently in awaf.toml:

[provider]
model = "claude-sonnet-4-5"

The default model (claude-haiku-4-5-20251001) is fast and cheap but occasionally produces invalid JSON on codebases with large artifact payloads. If you see this warning on more than one pillar per run, switching to Sonnet will resolve it.

Isolate the failing pillar

The default sequential mode already prints each pillar as it completes. Add a delay to slow things down further:

awaf run --delay 5

Score variability and apparent regressions

Symptom

Multiple pillars score the same value (e.g., six pillars all at 42), or a pillar score drops after a change unrelated to that pillar (e.g., Foundation drops after adding a runbook).

Cause: model behavior (score clustering)

All pillar evaluations run at temperature=0.0, which maximizes consistency but does not guarantee identical outputs — LLM providers may route requests across different hardware or model versions between runs. Scores are mostly deterministic: repeated runs on the same artifacts typically agree within ±3 points, but a single run is not a reliable point estimate.

Before treating a score as ground truth, sample at least 5–10 runs and record the mean and standard deviation. A score of 72 ± 2 is a stable reading; a score of 72 ± 15 is noise. The history command makes this easy:

# Run 5 times, then check history to see variance
for i in {1..5}; do awaf run; done
awaf history

Different models anchor at different values when evidence is incomplete (partial confidence):

• claude-haiku-4-5-20251001: clusters near 42
• claude-sonnet-4-6: clusters near 72

These are the models' holistic "partial credit" estimates, not computed scores. awaf v0.3.0+ addresses this with a mandatory tally field that forces mechanical per-criterion computation (see Dead letter detection below).

Cause: cross-pillar impression bleed

Every pillar receives the full artifact. If you add evidence that belongs to a different pillar (e.g., a runbook improves Op. Excellence), the model may slightly adjust its overall impression of the codebase, shifting unrelated pillar scores by ±5–10 points. awaf's pillar prompts instruct the model to score only within each pillar's domain, but smaller models are more susceptible to holistic reading.

Dead letter detection (v0.3.0+)

awaf-cli automatically detects suspect results and surfaces them for operator review:

• Known pathology scores (e.g., Haiku anchoring at 42) are flagged.
• Output truncation: if a pillar's response was cut off mid-stream (output tokens near the provider's max_tokens limit), the result is flagged rather than silently scored as 0.
• Score clustering: if 3 or more pillars return the same integer score, the cluster is flagged as possible model anchoring.

Suspect pillars are shown in a SUSPECT RESULTS block in the output and marked with ! in the pillar table. They are included in the overall score — suspect is a warning for operators to review, not a veto that silently drops pillars from the denominator. The run still completes; suspect results are visible so you can decide whether to re-run with a stronger model or accept the result.

What to do

• Don't chase single-run regressions. A 5-point drop after an unrelated change is noise. Track band changes, not point changes. See Scoring and Bands.
• Use --runs N to confirm a band change is real. If all N runs land in the same new band, it is a real regression. If they split across two bands, it is variance.

  awaf run --runs 3 --force
  

• Use a stronger model for stable scoring. Sonnet and Opus show much less clustering and cross-pillar bleed than Haiku:

  awaf run --model claude-sonnet-4-6
  

• Gate CI on band, not points. Regression detection now fires only on band drops, not point drops.
• Run sequentially (default). Sequential execution ensures Foundation writes the artifact cache before pillars 2–10 run, maximizing cache hits. --parallel may cause multiple cache writes if requests race before Foundation completes.

Contributing

Bug reports, feature requests, and PRs welcome. Provider adapter contributions especially welcome — see PROVIDER_SPEC.md for the interface contract.

For changes to the AWAF specification itself (pillar definitions, scoring questions, methodology), open an issue in the AWAF specification repo. This repo is for the implementation.

License

Apache 2.0. See LICENSE.

Related

• YogirajA/AWAF: The AWAF open specification
• PROVIDER_SPEC.md: Provider abstraction layer spec — build your own adapter
• Are We Building AI Agents Like We Built Microservices?: The post that introduced AWAF