crilio

The CI/CD quality gate for AI — pytest for prompts.

If Crilio helps you, please ⭐ star this repo — it helps other devs find it and motivates me to keep building.

☁️ Join the Crilio Cloud Waitlist (Get 40% off team dashboards & analytics when we launch)

Crilio stops prompt regressions from reaching production. Version your prompts + natural-language rules in crilio.yaml — Crilio calls your model, judges every response with an LLM, and reports PASS / FAIL per rule.

✨ Features

• LLM-as-a-Judge — Strict Pydantic-verified verdicts (rule_passed: bool), temp 0. No flaky free-text parsing.
• BYOK — Your keys, your bill. OpenAI + Anthropic. Typically < $0.01 / test on gpt-4o-mini.
• CI/CD Native — exit 1 blocks PRs in GitHub Actions. Locally it warns but never blocks.
• PR Comments — Auto-posts formatted failure details to the PR when running in Actions (GITHUB_TOKEN, silent fail, never blocks gate).
• Leak Guard — Rejects crilio.yaml containing sk-.../api_key — keys must be in .env/Secrets, never committed.
• Local Bots — target: {command: "python bot.py '{{prompt}}'"} runs any local model (Ollama/vLLM) via stdout → Judge, $0 target.
• Telemetry — Anonymous cli_command/cli_run events to PostHog (no prompts/keys). Disable via --off-tracking, CRILIO_DISABLE_TELEMETRY=1/DO_NOT_TRACK=1, or settings.telemetry: false.
• Skip & List — skip: true per-test to pause, crilio ls [--tag] [--json] to preview tests without running.
• Budget Guard — max_monthly_budget_usd halts the run when cost exceeds cap. Delete the line or leave it blank for unlimited.

🚀 Quick Start

pip install crilio

export OPENAI_API_KEY="sk-proj-..."  # or ANTHROPIC_API_KEY — .env also works
crilio init                          # creates crilio.yaml
crilio run --dry-run                 # validate without API calls
crilio run                           # Target → Judge → gate

crilio with no args shows status, budget, and next steps.

⚙️ Usage

Provider is inferred from env if not set in crilio.yaml. Keys are never flags — env only.

settings:
  target_model: gpt-4o
  judge_model: gpt-4o-mini
  max_monthly_budget_usd: 10.0  # delete or leave blank for unlimited

tests:
  - name: Refund Policy Check
    prompt: How long do I have to return a product?
    rules:
      - Must mention the 30-day return window.
      - Must NOT mention competitor names.
    tags: ["smoke", "critical"]

  - name: JSON Format Check
    prompt: |
      Return ONLY this JSON and nothing else: {"status": "shipped", "order_id": "12345"}
    rules:
      - Must return valid JSON with keys 'status' and 'order_id'.
      - Must NOT include apologies or extra prose.

  - name: Local Bot Check
    prompt: How long do I have to return a product?
    target:
      command: "python bot.py '{{prompt}}'"
    rules:
      - "Must mention the 30-day return window."
    tags: ["local"]

# 1. ollama serve & ollama pull llama3  (or mistral, qwen2, etc.)
# 2. crilio.yaml:
settings:
  target_model: gpt-4o
  judge_model: gpt-4o-mini

tests:
  - name: Ollama Refund
    prompt: "How long do I have to return a product?"
    target:
      command: "ollama run llama3 '{{prompt}}'"  # any model: mistral, qwen2, gemma
      # no placeholder also works → stdin: command: "ollama run llama3"
    rules:
      - "Must mention the 30-day return window."
    tags: ["ollama", "local"]

# 3. export OPENAI_API_KEY="sk-..."  # Judge still API
# 4. crilio run --tag ollama --verbose

🔄 CI/CD Integration

crilio init can scaffold this for you. Or drop in .github/workflows/crilio.yml:

name: Crilio AI Tests
on: [pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.10' }
      - run: pip install crilio
      - run: crilio run
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}  # for PR failure comments (auto-provided)

Add the secret matching your provider. A FAIL gate exits 1 and blocks the PR. On failure in Actions, Crilio posts a 🛑 Crilio AI Test Failed comment with test, rule, AI response and reason — requires permissions: pull-requests: write (or default GITHUB_TOKEN), fails silently locally or on API error and never blocks the gate.

📄 License

AGPL-3.0 — see LICENSE.