cylestio-perimeter 2.2.0

A configurable Python proxy server for LLM API requests with middleware support

Cylestio Gateway

A configurable Python proxy server for LLM API requests with interceptor support, built with FastAPI.

✨ Features

Core Functionality

• 🔄 LLM Provider Support: Proxy requests to OpenAI, Anthropic, and other LLM providers
• 📡 Streaming Support: Handle Server-Sent Events (SSE) for real-time responses
• 📊 Request Tracing: Capture and save request/response data to JSON files
• 🔍 Session Management: Intelligent session detection using message history hashing
• 🏷️ External ID Support: Custom session and agent IDs via x-cylestio-* headers
• ⚙️ Interceptor System: Extensible interceptors for tracing, logging, and analysis
• 💻 CLI Interface: Simple command-line interface with configuration file support
• 🐳 Docker Support: Ready-to-use Docker containers
• 📈 Metrics Endpoint: Monitor proxy performance and session statistics

Live Trace & Security

• 🖥️ Live Trace Dashboard: Real-time web UI for monitoring agent sessions and events
• 🔒 Security Analysis: OWASP LLM Top 10 pattern detection and risk scoring
• 🕵️ PII Detection: Automatic detection of personally identifiable information
• 🤖 MCP Integration: AI assistant integration via Model Context Protocol

Quick Start

Installation

1. Install through pip:

   pip install cylestio-perimeter
   

2. For Developers: Install directly from source code:

   git clone https://github.com/cylestio/cylestio-perimeter.git
   cd cylestio-perimeter
   
   # Create virtual environment
   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate
   
   # Install the package in development mode
   pip install -e .
   

3. Run the server:

   # With CLI arguments
   cylestio-perimeter run --base-url https://api.openai.com --type openai --api-key sk-your-key
   
   # With config file
   cylestio-perimeter run --config config.yaml
   

4. Or run with config file:

   python -m src.main --config config.yaml
   

Docker Usage

1. Using docker-compose (recommended):

   # Set environment variables
   export LLM_BASE_URL=https://api.openai.com
   export LLM_TYPE=openai
   export LLM_API_KEY=sk-your-key-here
   
   # Start the service
   docker-compose up -d
   

2. Using Docker directly:

   docker build -t llm-proxy .
   docker run -p 4000:4000 -e LLM_BASE_URL=https://api.openai.com -e LLM_TYPE=openai -e LLM_API_KEY=sk-your-key llm-proxy
   

Usage Examples

Basic Proxy Usage

# Start proxy server
cylestio-perimeter run --base-url https://api.openai.com --type openai --api-key sk-your-key

# Make requests to the proxy
curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Hello!"}]}'

Streaming Requests

curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Hello!"}], "stream": true}'

With Configuration File

# config.yaml
server:
  port: 4000
  host: "0.0.0.0"

llm:
  base_url: "https://api.openai.com"
  type: "openai"
  api_key: "sk-your-key-here"

interceptors:
  - type: "printer"
    enabled: true
    config:
      log_requests: true
      log_responses: true

External ID Headers

The gateway supports custom session and agent identification via headers:

# Custom session ID
curl -X POST http://localhost:4000/v1/chat/completions \
  -H "x-cylestio-session-id: my-session-123" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4", "messages": [{"role": "user", "content": "Hello!"}]}'

# Custom agent ID
curl -X POST http://localhost:4000/v1/chat/completions \
  -H "x-cylestio-agent-id: math-tutor-v2" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4", "messages": [{"role": "user", "content": "What is 2+2?"}]}'

# Both custom session and agent ID
curl -X POST http://localhost:4000/v1/chat/completions \
  -H "x-cylestio-session-id: user-session-456" \
  -H "x-cylestio-agent-id: customer-support-bot" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4", "messages": [{"role": "user", "content": "Help me reset my password"}]}'

See External Agent ID Documentation for complete details.

CLI Commands

The CLI supports several subcommands for different operations:

Run the Server

cylestio-perimeter run --base-url https://api.openai.com --type openai --api-key sk-your-key
cylestio-perimeter run --config config.yaml

Generate Example Config

cylestio-perimeter generate-config example.yaml

Validate Configuration

cylestio-perimeter validate-config config.yaml

Replay Recorded Traffic

cylestio-perimeter replay recordings/ --config config.yaml --delay 0.5

Get Help

cylestio-perimeter --help
cylestio-perimeter run --help

Development Mode

uvicorn src.main:app --reload --port 4000

Configuration

CLI Options

• --base-url: Base URL of target LLM API (required)
• --type: LLM provider type (required)
• --api-key: API key to inject into requests
• --port: Proxy server port (default: 4000)
• --host: Server host (default: 0.0.0.0)
• --log-level: Logging level (INFO, DEBUG, etc.)
• --config: Path to YAML configuration file

Interceptor Configuration

Available interceptors that can be enabled in your config file:

Interceptor	Description
printer	Logs requests/responses to console
message_logger	Saves LLM conversations to JSONL files
event_recorder	Records raw events per session
http_recorder	Records HTTP traffic for replay testing
cylestio_trace	Sends events to Cylestio platform
live_trace	Real-time dashboard with security analysis

Printer Interceptor

interceptors:
  - type: "printer"
    enabled: true
    config:
      log_requests: true
      log_responses: true
      log_body: false

Live Trace Interceptor

interceptors:
  - type: "live_trace"
    enabled: true
    config:
      server_port: 7100
      auto_open_browser: true
      storage_mode: "sqlite"  # "memory" (default) or "sqlite" for persistence
      enable_presidio: true   # PII detection

Testing

# Install dev dependencies
pip install -r requirements-dev.txt

# Run tests
pytest

# Run with coverage
pytest --cov=src

# Run specific tests
pytest tests/test_config.py -v

API Endpoints

• GET /health - Health check endpoint
• GET /metrics - Metrics endpoint with session statistics
• GET /config - Current server configuration and interceptor status
• /{path:path} - Catch-all proxy route (all HTTP methods)

Live Trace endpoints (when enabled):

• GET /api/dashboard - Dashboard data with agents and sessions
• GET /api/agent/{id} - Agent details with risk analysis
• GET /api/session/{id} - Session timeline with events
• POST /mcp - MCP protocol endpoint (SSE) for AI assistants

Session Management

The proxy includes intelligent session detection that tracks conversations across multiple requests:

• Hash-based Tracking: Uses message history hashing to identify unique conversations
• LRU Cache: Maintains up to 10,000 sessions with automatic eviction
• Session TTL: Sessions expire after 1 hour of inactivity
• Fuzzy Matching: Detects continued conversations even with slight variations
• Multiple Heuristics: Identifies new sessions based on message count and reset phrases

Session Configuration

session:
  enabled: true
  max_sessions: 10000
  session_ttl_seconds: 3600

Monitoring Sessions

Access session metrics via the /metrics endpoint:

curl http://localhost:4000/metrics

Response includes:

• Active sessions count
• Cache hit/miss rates
• Session creation rate
• Fuzzy match statistics

Live Trace Dashboard

When the live_trace interceptor is enabled, a real-time web dashboard is available for monitoring agent activity.

Features

• Real-time Monitoring: View active sessions and events as they happen
• Security Analysis: Automatic OWASP LLM Top 10 pattern detection
• PII Detection: Identify personally identifiable information in requests/responses
• Risk Scoring: Aggregate security scores for agents and sessions
• MCP Integration: AI assistants can query data via the /mcp endpoint

Quick Start

interceptors:
  - type: "live_trace"
    enabled: true
    config:
      server_port: 7100        # Dashboard port
      auto_open_browser: true  # Open browser on startup
      storage_mode: "sqlite"   # Persistent storage

Access the dashboard at http://localhost:7100 after starting the server.

Environment Variables

• LLM_BASE_URL - Base URL for LLM provider
• LLM_TYPE - LLM provider type
• LLM_API_KEY - API key for authentication
• LOG_LEVEL - Logging level (INFO, DEBUG, etc.)

Security

Cylestio Gateway implements comprehensive security measures to ensure safe deployment in enterprise environments.

Security Measures:

• Automated Vulnerability Scanning: Every release is scanned for known security issues
• Dependency Security: All third-party packages are continuously monitored for vulnerabilities
• Static Code Analysis: Source code is analyzed for security vulnerabilities using industry-standard tools
• Secret Detection: Pre-commit hooks prevent accidental credential exposure
• Supply Chain Security: Complete Software Bill of Materials (SBOM) provides full transparency
• License Compliance: Automated scanning ensures only approved licenses are used

Documentation: Security Policy

Development

Setting Up Development Environment

# Install development dependencies
pip install -r requirements-dev.txt

# Set up pre-commit hooks (includes security scanning)
pre-commit install

# Run tests with coverage
pytest --cov=src

# Run security checks locally
pre-commit run --all-files

Security Development Practices

1. Never commit secrets - Use environment variables for all credentials
2. Run pre-commit hooks - Automated security checks before each commit
3. Review security reports - Check CI security scan results
4. Follow secure coding - Follow standard Python security best practices

See CLAUDE.md for detailed development guidance and architecture information.

License

This project is developed according to the specifications in INSTRUCTIONS.md.