codejury 0.2.0

General-purpose Application Security AI audit framework -- five-layer architecture, capabilities as first-class data

codejury

A general-purpose Application Security AI audit framework. Domain knowledge (11 capabilities aligned with OWASP ASVS) lives in versioned YAML files as first-class data, keeping the framework core small.

The name comes from the core orchestration metaphor: code goes before a "jury" of adversarial roles -- Finder / Challenger / Judge -- that converge on a verdict.

Five-layer architecture

Layer 5  Task            task configuration (source + capabilities + orchestrator + agents)
Layer 4  Capability      YAML domain knowledge (authn / authz / input_validation ...)
Layer 3  Orchestrator    strategy (single / debate / pipeline / reflexion)
         Source          input (diff / function / repo)
         Agent           audit role (finder / challenger / judge / verifier)
Layer 2  Provider        model backend (anthropic / openai / litellm / mock)
Layer 1  Infrastructure  cross-cutting utilities (json parsing, ...)

Layers talk only through typed data. Each layer is an abstract base class (ABC) plus implementations, so the four axes (task / orchestration / model / input) compose independently.

Design notes

• Domain knowledge is data, not prompts: a capability YAML is readable by the LLM, by a rule engine, and by a human, and is versioned alongside code.
• Explains both "why it's wrong" and "why it's fine": every capability yields a Verdict, recording safe matches too -- a checkup dimension rather than an anomaly filter.

Status

Usable end to end across all five layers:

• Orchestrators: single, pipeline, debate, reflexion
• Sources: diff, function, repo (with chunking)
• Providers: anthropic, openai, litellm, mock (plus an opt-in retry wrapper)
• Capabilities: all 11 OWASP ASVS areas
• Tasks: named presets in tasks/ (e.g. audit_diff_debate)
• Reporting: text, markdown, json
• Evaluation: a golden-case precision/recall harness

The golden set ships with seed cases; real precision/recall numbers need a model (codejury eval with a provider key).

Install

pip install codejury                 # core + CLI
pip install 'codejury[anthropic]'    # add the provider you'll use (anthropic / openai / litellm)

Usage

A real audit calls a model, so set the provider's key first (see .env.example):

export ANTHROPIC_API_KEY=sk-ant-...   # or OPENAI_API_KEY for --provider openai

# Audit a unified diff against the capability library
git diff | codejury audit --orchestrator debate --provider anthropic --format markdown -

# Run a named task preset (tasks/*.yaml)
git diff | codejury run audit_diff_debate -

# Score detection quality against the golden cases
codejury eval --provider anthropic

# Through a LiteLLM proxy / gateway. The flags default to CODEJURY_API_BASE /
# CODEJURY_API_KEY / CODEJURY_MODEL, so with those in a sourced .env this is just:
#   codejury audit --provider litellm -
git diff | codejury audit --provider litellm \
  --api-base https://litellm.example.com --api-key "$LITELLM_KEY" --model your-alias -

# No API key needed: prove the pipeline composes with mock layers
codejury dry-run

audit and run read a diff from a file argument or stdin (-). The provider key is read from the environment: ANTHROPIC_API_KEY for --provider anthropic, OPENAI_API_KEY for --provider openai. Without a key the model providers raise an authentication error; codejury dry-run needs no key.

A task YAML can pin the provider, model, and base URL (the key stays in the environment), so codejury run works through a proxy too:

# mytasks/proxy_scan.yaml -> codejury run proxy_scan --tasks mytasks
name: proxy_scan
orchestrator: debate
provider: litellm
model: your-alias
api_base: https://litellm.example.com   # key from CODEJURY_API_KEY

Development

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest