llm-sniffer 0.4.0

LLM Sniffer - OpenAI-compatible reverse proxy with request/response inspector

LLM Sniffer

LLM Sniffer is an OpenAI-compatible reverse proxy with request/response inspector.

Features

• Reverse Proxy: OpenAI-compatible API proxy
• Request/Response Inspector: Monitor and inspect all LLM requests and responses
• SSE Support: Server-Sent Events for streaming responses
• Modern UI: Clean web interface for inspecting traffic
• Multi-Upstream Support: Configure multiple LLM backends
• Dynamic Configuration: Switch between upstreams via command line or config
• Trace Save: Save request/response JSON to files for debugging and analysis

Installation

pip install llm-sniffer

Quick Start

Start the proxy server with default settings:

llm-sniffer

Configure your LLM client to use:

http://127.0.0.1:7654/v1

Then open http://127.0.0.1:7655 in your browser to inspect requests.

Command Line Options

Proxy Server Options

llm-sniffer [OPTIONS]

Options:
  --upstream-url URL     Direct upstream URL (highest priority)
  --upstream-name NAME   Use upstream from configuration file
  --upstream NAME/URL    Upstream name or URL (deprecated)
  --proxy-port PORT      Proxy service port (default: 7654)
  --ui-port PORT         UI service port (default: 7655)
  --max-records N        Maximum number of records to keep (default: 200)
  --think on|off         Enable/disable thinking mode (default: on)
  --host ADDRESS         Bind address (default: 127.0.0.1)
  --params JSON          Parameters to inject into each request body
  --save-trace           Enable saving request/response traces to files
  --output-dir DIR       Directory to save trace files (default: llm_sniffer_trace)

Configuration Management

# Initialize default config file
llm-sniffer config init

# List all configured upstreams
llm-sniffer config list

# Set active upstream
llm-sniffer config set kimi

# Add new upstream
llm-sniffer config add myserver --url http://localhost:8080 --description "My Server"

# Remove upstream
llm-sniffer config remove myserver

# Print config file path
llm-sniffer config path

Configuration File

Default config path: ~/.llm_sniffer/config.yaml

upstreams:
  local:
    url: http://127.0.0.1:8000
    api_key: ""
    description: Local LLM server (vLLM, Ollama, etc.)
  openai:
    url: https://api.openai.com
    api_key: ""
    description: OpenAI API
  qwen:
    url: https://dashscope.aliyuncs.com/compatible-mode
    api_key: ""
    description: Qwen (Alibaba Cloud)
  kimi:
    url: https://api.moonshot.cn
    api_key: ""
    description: Kimi (Moonshot AI)

active_upstream: local
proxy_port: 7654
ui_port: 7655
max_records: 200
think: on
host: 127.0.0.1
save_trace: false
output_dir: llm_sniffer_trace

Upstream Selection Priority

1. --upstream-url (command line) - Highest priority
2. --upstream-name (command line)
3. --upstream (command line) - Deprecated
4. active_upstream in config file - Lowest priority

Trace Save

Enable trace save to save request/response JSON files for debugging and analysis.

Usage

# Enable trace save with default output directory
llm-sniffer --save-trace

# Enable trace save with custom output directory
llm-sniffer --save-trace --output-dir /path/to/traces

Output Structure

When trace save is enabled, files are saved in the following structure:

llm_sniffer_trace/
└── 20240101_120000_abc12345/
    ├── 0001_8f3d2a1b/
    │   ├── request.json
    │   └── response.json
    ├── 0002_9e4c3b2c/
    │   ├── request.json
    │   └── response.json
    └── ...

• Each session gets its own directory with timestamp and session ID
• Each request gets a numbered subdirectory
• request.json: The request body sent to the upstream
• response.json: The response body received from the upstream (merged for streaming)

Configuration

You can also configure trace save in the config file:

save_trace: true
output_dir: /my/custom/trace/path

Examples

# Use specific upstream URL
llm-sniffer --upstream-url http://localhost:8080

# Use upstream from config
llm-sniffer --upstream-name kimi

# Custom ports
llm-sniffer --proxy-port 8080 --ui-port 8081

# Start with custom upstream and inject parameters
llm-sniffer --upstream-name openai --params '{"temperature": 0.7}'

# Enable trace save
llm-sniffer --save-trace

# Enable trace save with custom output directory
llm-sniffer --save-trace --output-dir ./traces

License

MIT License