sibylline-skald 0.1.0

In-process adapter library for Python MCP servers that enables structured feedback collection on tool usefulness, clarity, and fit

Skald - MCP Feedback Adapter

"A storyteller for your tools' performance, letting them speak back through AI agents when they shine or stumble."

Skald is an in-process adapter library for Python MCP servers that enables structured feedback collection on tool usefulness, clarity, and fit from AI agents.

Features

• Transparent Tool Wrapping: Drop-in wrapper that intercepts MCP tool calls and adds trace IDs with metrics collection
• Structured Feedback Collection: New feedback.report tool with strict JSON schema validation
• Smart Feedback Invitations: Automatically invites feedback based on errors, latency, and output size
• Opt-out Controls: Decorator and context manager for suppressing feedback collection
• Multiple Storage Backends: SQLite default with optional Parquet export and data redaction
• Multiple Transports: Support for stdio, TCP, and Unix socket transports

Installation

pip install skald

Quick Start

Basic Usage

import skald
from your_mcp_server import YourMCPServer

# Wrap your existing MCP server
upstream = YourMCPServer()
survey = skald.SurveyingProxy(
    upstream=upstream,
    store="sqlite:///feedback.db",
    invite_policy={"error": True, "p95_ms": 5000, "large_output_kb": 256},
    sample_neutral=0.10,
    ttl_hours=24
)

# Your server now collects feedback transparently

Using Opt-out Controls

# Opt out specific functions
@survey.opt_out(reason="contains sensitive data")
def run_sensitive_query(sql: str) -> dict:
    return execute_query(sql)

# Suppress collection for a block of code
with survey.suppressed(reason="benchmark"):
    for _ in range(1000):
        call_some_tool()

Running a Server

from skald.transport import TCPTransport

# Create transport and serve
transport = TCPTransport(survey)
await transport.serve(host="0.0.0.0", port=8765)

CLI Usage

Skald includes a CLI for running demo servers and querying feedback data:

# Run over TCP
skald tcp --host localhost --port 8765

# Run over stdio  
skald stdio

# Query stored feedback
skald query --store sqlite:///feedback.db --limit 10

# Clean up expired data
skald cleanup --store sqlite:///feedback.db --ttl-hours 24

Agent Integration

To collect feedback from AI agents, add this line to your system prompt:

When a tool provides a trace_id in its response metadata and invite_feedback=true, you may optionally use feedback.report to provide structured feedback on the tool's helpfulness (1-5), fit (1-5), clarity (1-5), confidence (0.0-1.0), and up to 3 short suggestions for improvement.

Feedback Schema

The feedback.report tool accepts this JSON schema:

{
  "trace_id": "550e8400-e29b-41d4-a716-446655440000",
  "helpfulness": 4,
  "fit": 3, 
  "clarity": 5,
  "confidence": 0.8,
  "better_alternative": "different_tool",
  "suggestions": ["Try tool X", "Use parameter Y"],
  "notes": "This worked well overall"
}

Fields

• trace_id (required): UUID from the tool response metadata
• helpfulness (required): How helpful was the tool (1-5)
• fit (required): How well did the tool fit the task (1-5)
• clarity (required): How clear was the tool output (1-5)
• confidence (required): Confidence in this feedback (0.0-1.0)
• better_alternative (optional): Enum suggesting better alternatives
• suggestions (optional): Up to 3 suggestions, each ≤100 characters
• notes (optional): Additional notes

Configuration

Invite Policy

Control when feedback is invited:

invite_policy = {
    "error": True,           # Invite on errors
    "timeout": True,         # Invite on timeouts  
    "p95_ms": 5000.0,       # Latency threshold (ms)
    "large_output_kb": 256.0 # Output size threshold (KB)
}

Storage Configuration

# SQLite (default)
store = "sqlite:///path/to/feedback.db"

# Custom storage backend
from skald.storage.sqlite import SQLiteStorage
storage = SQLiteStorage("/custom/path.db")
survey = SurveyingProxy(upstream, store=storage)

Data Redaction

Configure custom data redaction:

def custom_redactor(args: dict) -> dict:
    """Remove sensitive data from tool arguments."""
    redacted = args.copy()
    if 'password' in redacted:
        redacted['password'] = '[REDACTED]'
    return redacted

survey = SurveyingProxy(
    upstream=upstream,
    redactor=custom_redactor
)

Storage Schema

Skald uses two main tables:

tool_runs

• trace_id (PK): UUID trace identifier
• timestamp: Execution timestamp
• agent_id: ID of the calling agent
• tool_name: Name of the tool called
• status: success/error/timeout
• latency_ms: Execution latency in milliseconds
• output_bytes: Size of output in bytes
• invite_feedback: Whether feedback was invited
• opt_out: Whether collection was opted out
• args_redacted: Redacted tool arguments (JSON)

tool_feedback

• trace_id (FK): Reference to tool_runs
• agent_id: ID of the agent providing feedback
• helpfulness: Helpfulness rating (1-5)
• fit: Fit rating (1-5)
• clarity: Clarity rating (1-5)
• confidence: Confidence score (0.0-1.0)
• better_alternative: Better alternative suggestion
• suggestions: List of suggestions (JSON)
• notes: Additional notes
• valid: Whether feedback passed validation
• raw_json: Raw feedback JSON
• timestamp: Feedback timestamp

Architecture

Skald follows a clean architecture with these components:

• Core: SurveyingProxy - Main proxy class that wraps MCP servers
• Schema: Pydantic models for data validation
• Storage: Pluggable storage backends (SQLite default)
• Transport: Communication protocols (stdio, TCP, Unix sockets)
• Decorators: Opt-out mechanisms (@opt_out, with suppressed())

Development

Setup

git clone https://github.com/sibyllinesoft/skald.git
cd skald
uv venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
uv pip install -e ".[dev,test]"

Testing

pytest
pytest --cov=skald --cov-report=html

Code Quality

black skald tests
isort skald tests  
mypy skald
ruff skald tests

License

MIT License - see LICENSE file for details.

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.