phylax 1.5.0

CI-native regression enforcement for LLM outputs.

Phylax — CI for AI Behavior

Stop AI regressions before they reach production.

---

⚡ 30-Second Example

pip install phylax

from phylax import trace, expect

@trace(provider="openai")
@expect(must_include=["refund"], must_not_include=["lawsuit"])
def reply(prompt):
    return llm(prompt)  # your LLM call

phylax check

❌ FAIL — expected "refund" in response
   Trace ID: abc-123
   Violation: must_include rule failed

That's it. Your CI now blocks AI behavior regressions.

---

🎯 What Phylax Does

Phylax enforces deterministic contracts on LLM outputs. When your AI's behavior changes across model versions, prompts, or configurations — Phylax catches it.

Developer writes rules   →   Phylax tests every response   →   CI blocks regressions

Phylax is NOT monitoring, observability, or AI-based evaluation. It's a test framework for AI behavior.

---

🚀 Quick Start

1. Install

pip install phylax[openai]   # or phylax[google], phylax[groq], phylax[all]

2. Write Traced Code

from phylax import trace, expect, execution, OpenAIAdapter

# Single call with expectations
@trace(provider="openai")
@expect(must_include=["refund"], max_latency_ms=3000)
def handle_refund(message: str) -> str:
    adapter = OpenAIAdapter()
    response, _ = adapter.generate(prompt=message)
    return response

# Multi-step agent flow
@trace(provider="openai")
@expect(must_include=["intent"])
def classify(message: str) -> str:
    adapter = OpenAIAdapter()
    response, _ = adapter.generate(prompt=f"Classify: {message}")
    return response

# Track agent workflows as execution graphs
with execution() as exec_id:
    intent = classify("I want a refund")
    response = handle_refund("Process refund for order #123")

3. Enforce in CI

# .github/workflows/phylax.yml
name: Phylax CI
on: [push, pull_request]
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install phylax[openai]
      - run: phylax check    # exits 1 on contract violation
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

---

📋 Expectation Rules

Rule	What It Enforces	Example
must_include	Response contains required text	["refund", "policy"]
must_not_include	Response excludes forbidden text	["lawsuit", "attorney"]
max_latency_ms	Response time limit	3000 (3 seconds)
min_tokens	Minimum response length	50 tokens

Combine with Logic

from phylax import trace, expect, AndGroup, OrGroup, MustIncludeRule, MaxLatencyRule

@trace(provider="openai")
@expect(rules=AndGroup([
    MustIncludeRule("refund"),
    OrGroup([
        MaxLatencyRule(3000),
        MustIncludeRule("processing"),
    ]),
]))
def handle(message):
    ...

---

🔬 Surface Enforcement

Enforce contracts on any LLM output type — not just text.

from phylax import (
    Surface, SurfaceEvaluator,
    FieldExistsRule,             # JSON field must exist
    TypeEnforcementRule,          # Strict type checking
    ToolPresenceRule,             # Tool must be called
    StepCountRule,                # Execution must have N steps
    ExactStabilityRule,           # Output must not change between runs
)

Surface Type	Rules	Use Case
Structured Output	Field exists, type, value, enum, array bounds	JSON API responses
Tool Calls	Presence, count, argument, ordering	Agent tool usage
Execution Traces	Step count, forbidden transitions, required stages	Multi-step workflows
Cross-Run Stability	Exact match, allowed drift	Regression detection

---

📊 Execution Graphs

Visualize and debug multi-step agent workflows:

from phylax import trace, expect, execution

with execution() as exec_id:
    step1 = classify(message)      # → Node 1
    step2 = research(step1)        # → Node 2
    step3 = respond(step2)         # → Node 3

# Phylax builds a DAG:
#   [classify] → [research] → [respond]
#
# View it:
#   phylax server → http://127.0.0.1:8000/ui

---

🛡️ Enforcement Modes

from phylax import ModeHandler, EnforcementMode

handler = ModeHandler(mode=EnforcementMode.ENFORCE)
# enforce    → CI fails on violation (default)
# quarantine → CI fails, but logs for review
# observe    → CI passes, violations logged only

---

🔧 Commands

Command	What It Does
phylax init	Initialize config
phylax server	Start API + UI
phylax list	List traces (--failed for failures only)
phylax show <id>	Inspect a trace
phylax bless <id>	Mark as golden baseline
phylax check	CI enforcement — exits 1 on violation
phylax --version	Show version

---

🔌 Supported Providers

pip install phylax[all]   # Install all providers

Provider	Import	Env Variable
OpenAI	from phylax import OpenAIAdapter	OPENAI_API_KEY
Gemini	from phylax import GeminiAdapter	GOOGLE_API_KEY
Groq	from phylax import GroqAdapter	GROQ_API_KEY
Mistral	from phylax import MistralAdapter	MISTRAL_API_KEY
HuggingFace	from phylax import HuggingFaceAdapter	HF_TOKEN
Ollama	from phylax import OllamaAdapter	OLLAMA_HOST

---

📁 Examples & Templates

Examples — Learn by doing

Example	What You'll Learn
examples/quickstart/	Basic tracing in 10 lines
examples/support_bot/	Content safety enforcement
examples/summarization/	Latency + quality contracts
examples/agent_workflow/	Multi-step agents with execution graphs

Templates — Start building

Template	What It Includes
templates/ai-chatbot/	Chatbot + contracts + CI config
templates/ai-agent/	Multi-step agent + execution tracking
templates/ai-rag/	RAG pipeline + grounding enforcement

---

📖 Documentation

Start here:

• Quickstart — 10 minutes to CI enforcement
• Mental Model — What Phylax is and isn't
• Providers — LLM provider reference
• Error Codes — Error code reference

Reference:

• API Contract — Stability guarantees
• Execution Context — Trace grouping
• Non-Goals — What Phylax will never do
• Versioning — Release policy

Advanced:

• Surface Enforcement
• Graph Model
• Failure Playbook
• Performance
• Invariants

---

🏛️ Architecture

phylax/
├── _internal/
│   ├── expectations/    # Deterministic rule engine
│   ├── surfaces/        # Surface enforcement (JSON, tools, traces)
│   ├── metrics/         # Expectation health & coverage
│   ├── modes/           # Enforcement mode control
│   ├── meta/            # Dilution guards
│   ├── artifacts/       # Machine-consumable CI outputs
│   ├── adapters/        # LLM provider adapters
│   └── graph.py         # Execution graphs
├── cli/                 # CLI commands
├── server/              # API + health routes
└── ui/                  # Web inspector

799 tests · v1.4.1 · All 4 axes complete

---

TL;DR

Phylax is CI for AI behavior. It records LLM outputs, evaluates them against declared contracts, and fails builds when behavior regresses. No scoring. No judgment. No AI-based evaluation. Just deterministic enforcement.

---

License

MIT License

Made with love❤️