🛡️ AI Failure Observatory

A lightweight, local-first platform and Model Context Protocol (MCP) Server designed to detect, track, stress-test, and report on 6 critical LLM behavioral failure modes before deploying models into mission-critical production workflows.

Built with Zero External Core Dependencies using Python's standard library (http.server, urllib.request, json, sqlite3/file persistence) with optional FastMCP connectivity for autonomous AI safety agents.

📸 Visual Showcase

🏗️ Architecture Overview

flowchart TD
    subgraph PresentationLayer["🖥️ Presentation & Client Interfaces"]
        WebUI["Single-Page Dashboard (index.html)"]
        Bilingual["TR ⟷ EN Internationalization Engine"]
        ReportExport["Markdown / JSON Audit Exporter"]
    end

    subgraph DualAccessLayer["⚡ Dual-Mode Core Interfaces"]
        HTTPServer["Web Dashboard (server.py on Port 5089+)"]
        MCPServer["Model Context Protocol Server (mcp_server.py)"]
    end

    subgraph AnalysisEngine["🧠 Behavioral Safety Analyzers"]
        FailureAnalyzer["src/failure_analyzer.py
(Heuristic & Pattern Detection)"]
        RiskEngine["analysis/risk_analysis.py
(Composite Risk Index Calculation)"]
        EvalRunner["experiments/reproducible_evals/run_all_evals.py
(Automated Benchmarks)"]
    end

    subgraph ModelLayer["🤖 Multi-Provider LLM Gateway"]
        HeuristicSim["Built-in Heuristic Simulator (Offline)"]
        Gemini["Google Gemini (2.0 Flash / 1.5 Pro)"]
        OpenAI["OpenAI (GPT-4o / o3-mini)"]
        Claude["Anthropic Claude (3.5 Sonnet)"]
    end

    subgraph AIAssistants["🤖 AI Safety Agents"]
        ClaudeDesktop["Claude Desktop"]
        CursorIDE["Cursor IDE"]
        Antigravity["Google Antigravity"]
    end

    PresentationLayer <--> HTTPServer
    MCPServer <==> AIAssistants
    DualAccessLayer <--> AnalysisEngine
    AnalysisEngine <--> ModelLayer
    AnalysisEngine <--> StorageLayer["💾 Persistent Incident Storage (data/incidents.json)"]

🔌 Model Context Protocol (MCP) Server

AI Failure Observatory acts as an autonomous AI Safety & Red-Teaming Inspector over MCP. AI assistants in Claude Desktop, Cursor, VS Code, or Antigravity can audit generated text for safety violations, evaluate prompts for vulnerabilities, query formal risk taxonomies, and execute benchmark evaluations without opening a browser.

🛠️ Exposed MCP Tools

🚀 Claude Desktop & Cursor Setup

Add the following to your claude_desktop_config.json or Cursor MCP settings:

{
  "mcpServers": {
    "ai-failure-observatory": {
      "command": "uvx",
      "args": ["ai-failure-observatory"]
    }
  }
}

💡 Example AI Prompts with MCP

Once connected, ask your AI assistant:

• "Audit this model response for invented citations or academic hallucination."
• "What are the product mitigations for fake confidence and calibration errors according to the failure taxonomy?"
• "Scan this 5-turn conversation transcript for context loss and negative instruction drift."
• "Run the benchmark evaluation suite and give me the pass/fail score across all failure modes."

🔬 Formal AI Failure Taxonomy

The platform models failures across two primary axes defined in taxonomy/ai_failure_taxonomy.md:

🛠️ Quick Start

1. Zero-Install Execution via uvx

# Launch the Web Dashboard (Port 5089)
uvx ai-failure-observatory --web

# Launch MCP Stdio Server (for AI agents)
uvx ai-failure-observatory

2. Standard Installation via pip

pip install ai-failure-observatory
ai-failure-observatory --web

3. Local Development & Testing

git clone https://github.com/adacreativeco/ai-failure-observatory.git
cd ai-failure-observatory
python server.py

Open http://localhost:5089 in your browser.

# Run automated tests
python -m unittest discover tests

# Run benchmark suite
python experiments/reproducible_evals/run_all_evals.py

📂 Project Structure

ai-failure-observatory/
├── server.py                       # Zero-dependency HTTP server & REST API
├── mcp_server.py                   # Model Context Protocol (MCP) server entry point
├── index.html                      # Dark-glassmorphic SPA dashboard
├── pyproject.toml                  # Standard Python packaging metadata
├── requirements.txt                # Optional dependencies
├── analysis/
│   ├── risk_analysis.py            # Composite risk calculation & report generator
│   └── reports/                    # Generated compliance reports (JSON/Markdown)
├── src/
│   ├── failure_analyzer.py         # 6 Core behavioral heuristic analyzers
│   ├── mcp_tools.py                # MCP safety inspection & auditing tools
│   ├── llm_client.py               # Multi-provider LLM connector (Gemini, Claude, OpenAI)
│   ├── storage.py                  # JSON incident storage engine
│   └── utils.py                    # Formatter & helper utilities
├── taxonomy/
│   ├── ai_failure_taxonomy.md      # Formal failure mode taxonomy
│   └── taxonomy_utils.py           # Taxonomy parser and validator
├── experiments/
│   ├── reproducible_evals/         # Automated reproducible benchmark tests
│   │   ├── run_all_evals.py        # Master test runner
│   │   ├── test_hallucination_citation.py
│   │   ├── test_fake_confidence.py
│   │   ├── test_context_loss.py
│   │   ├── test_instruction_drift.py
│   │   ├── test_manipulation.py
│   │   └── test_recursive_collapse.py
│   └── synthetic/                  # Synthetic test data generators
└── tests/
    ├── test_observatory.py         # Core observatory unit test suite
    └── test_mcp.py                 # MCP tools & server test suite (21 tests total)

📄 License

Distributed under the Apache 2.0 License. See LICENSE for details.