foil-sdk 0.5.0

Foil SDK for monitoring and logging AI model invocations

Foil Python SDK

Python SDK for monitoring and logging AI agent invocations with Foil. Features distributed tracing, semantic search, custom evaluations, multimodal content support (images, documents), signals/feedback, and OpenAI integration.

Installation

pip install foil-sdk

Or install from source:

pip install -e /path/to/foil-sdk

Quick Start

from foil import Foil, create_foil_tracer, SpanKind

# Create a tracer for your agent
tracer = create_foil_tracer(
    api_key="your-api-key",
    agent_name="my-agent",
    base_url="https://api.foil.dev/api",  # or your self-hosted URL
)

# Trace an agent execution
async def my_agent():
    async with tracer.trace("weather-query", input="User asked about weather") as ctx:
        # Create an LLM span
        span = await ctx.start_span(SpanKind.LLM, "gpt-4", input="What is the weather?")

        # ... call your LLM ...

        await span.end(
            output="The weather is sunny today!",
            tokens={"prompt": 10, "completion": 8, "total": 18}
        )

        return "Agent completed"

Table of Contents

• Distributed Tracing
• Semantic Search
• Custom Evaluations
• Multimodal Content
• Signals & Feedback
• OpenAI Integration
• Experiments (A/B Testing)
• Low-Level API
• API Reference
• Examples

Distributed Tracing

Using the Tracer

from foil import create_foil_tracer, SpanKind

tracer = create_foil_tracer(
    api_key="your-api-key",
    agent_name="customer-support-agent",
)

async with tracer.trace("support-query", input=user_message, session_id=conversation_id) as ctx:
    # LLM span for the main model call
    span = await ctx.start_span(SpanKind.LLM, "gpt-4-turbo", input=messages)

    response = await openai.chat.completions.create(
        model="gpt-4-turbo",
        messages=messages,
    )

    await span.end(
        output=response.choices[0].message.content,
        tokens={
            "prompt": response.usage.prompt_tokens,
            "completion": response.usage.completion_tokens,
            "total": response.usage.total_tokens,
        }
    )

Span Kinds

from foil import SpanKind

SpanKind.AGENT      # Root agent span
SpanKind.LLM        # Language model calls
SpanKind.TOOL       # Tool/function executions
SpanKind.CHAIN      # Chain of operations
SpanKind.RETRIEVER  # RAG retrieval operations
SpanKind.EMBEDDING  # Embedding model calls
SpanKind.CUSTOM     # Custom operation types

Nested Spans

async with tracer.trace("order-lookup") as ctx:
    # LLM decides to use a tool
    llm_span = await ctx.start_span(SpanKind.LLM, "gpt-4", input="Find order #12345")

    # Tool execution (child of LLM span)
    tool_span = await ctx.start_span(SpanKind.TOOL, "lookup_order", input={"order_id": "12345"})
    result = await database.find_order("12345")
    await tool_span.end(output=result)

    await llm_span.end(output=f"Order found: {result}")

Semantic Search

Search through your traces using natural language queries. Foil automatically embeds your trace content and enables AI-powered semantic search.

Basic Search

from foil import Foil

foil = Foil(
    api_key="your-api-key",
    base_url="https://api.foil.dev/api",
)

# Search for conversations about a topic
results = foil.semantic_search("conversations about refund requests")

print(f"Found {len(results['results'])} matching traces")
for result in results["results"]:
    print(f"Trace: {result['traceId']}, Similarity: {result['similarity'] * 100:.1f}%")

Search with Filters

results = foil.semantic_search(
    "user asking about pricing",
    agent_id="customer-support-agent",  # Filter by specific agent
    agent_name="support-bot",           # Or filter by agent name
    from_date="2024-01-01",             # Start date (ISO format)
    to_date="2024-12-31",               # End date (ISO format)
    limit=10,                           # Max results (default: 20)
    offset=0,                           # Pagination offset
    threshold=0.4,                      # Min similarity score 0-1 (default: 0.3)
)

Find Similar Traces

Find traces that are semantically similar to a specific trace:

# Find traces similar to a known trace
similar = foil.find_similar_traces(
    "trace-abc-123",
    limit=5,
    threshold=0.4,
)

print(f"Found {len(similar['results'])} similar traces")
for result in similar["results"]:
    print(f"{result['traceId']}: {result['similarity'] * 100:.1f}% similar")

Check Semantic Search Status

# Get overall embedding status
status = foil.get_semantic_search_status()
print(f"Embedded: {status['embeddedSpans']} / {status['totalSpans']} spans")
print(f"Coverage: {status['coveragePercent']}%")
print(f"Ready: {status['ready']}")

# Get status for specific agent
agent_status = foil.get_semantic_search_status("my-agent-id")

Custom Evaluations

Define custom evaluation criteria to analyze your agent's responses. Evaluations run automatically on new traces and can be boolean, score-based, or categorical.

Get Evaluation Templates

# List available pre-built evaluation templates
templates = foil.get_evaluation_templates()

for t in templates.get("templates", []):
    print(f"{t['name']}: {t.get('description', 'N/A')}")

Create a Boolean Evaluation

evaluation = foil.create_evaluation("agent-123", {
    "name": "professional_tone",
    "description": "Checks if the response maintains a professional tone",
    "prompt": """Evaluate the assistant's response for professional tone.

Consider:
- Is the language respectful and courteous?
- Does it avoid slang or casual expressions?
- Is it helpful without being condescending?

Return true if professional, false otherwise.""",
    "evaluationType": "boolean",
    "enabled": True,
})

print(f"Created evaluation: {evaluation['evaluation']['name']}")

Create a Score Evaluation (1-10)

evaluation = foil.create_evaluation("agent-123", {
    "name": "helpfulness_score",
    "description": "Rates response helpfulness on a scale of 1-10",
    "prompt": """Rate the helpfulness of the response on a scale of 1-10.

Scoring:
- 1-3: Unhelpful
- 4-5: Somewhat helpful
- 6-7: Helpful
- 8-9: Very helpful
- 10: Exceptional

Return a single number from 1 to 10.""",
    "evaluationType": "score",
    "scoreMin": 1,
    "scoreMax": 10,
    "enabled": True,
})

Create a Category Evaluation

evaluation = foil.create_evaluation("agent-123", {
    "name": "response_intent",
    "description": "Categorizes the type of response",
    "prompt": """Classify the response into one category:

- informational: Provides facts or explanations
- action: Guides through a process
- clarification: Asks for more information
- escalation: Suggests human assistance
- closure: Wraps up the conversation

Return only the category name.""",
    "evaluationType": "category",
    "categories": ["informational", "action", "clarification", "escalation", "closure"],
    "enabled": True,
})

Test an Evaluation

Validate your evaluation prompt before enabling:

result = foil.test_evaluation("agent-123", evaluation_id, {
    "input": "How do I reset my password?",
    "output": "I'd be happy to help! Go to Settings > Security > Reset Password.",
})

print(f"Result: {result['result']}")        # true/false, score, or category
print(f"Reasoning: {result['reasoning']}")  # Explanation

Add Few-Shot Examples

Improve evaluation accuracy with examples:

# Add a positive example
foil.add_evaluation_example("agent-123", evaluation_id, {
    "input": "Why is my order late?",
    "output": "I apologize for the delay. Let me check the status for you.",
    "expectedResult": True,
    "reasoning": "Professional and helpful response",
})

# Add a negative example
foil.add_evaluation_example("agent-123", evaluation_id, {
    "input": "Why is my order late?",
    "output": "idk check tracking yourself",
    "expectedResult": False,
    "reasoning": "Casual and unhelpful",
})

Manage Evaluations

# List all evaluations for an agent
evaluations = foil.get_agent_evaluations("agent-123")

# Get specific evaluation details
evaluation = foil.get_evaluation("agent-123", evaluation_id)

# Update an evaluation
foil.update_evaluation("agent-123", evaluation_id, {
    "description": "Updated description",
    "enabled": False,  # Disable temporarily
})

# Get evaluation analytics
analytics = foil.get_evaluation_analytics("agent-123", evaluation_id)
print(f"Total evaluations: {analytics.get('totalEvaluations', 0)}")
print(f"Distribution: {analytics.get('distribution')}")

# Clone a template
cloned = foil.clone_evaluation_template("agent-123", template_id)

# Remove an example
foil.remove_evaluation_example("agent-123", evaluation_id, example_id)

# Delete an evaluation
foil.delete_evaluation("agent-123", evaluation_id)

Multimodal Content

Foil supports multimodal input/output including images, documents, and other media types.

Media Categories

from foil import MediaCategory

MediaCategory.IMAGE       # Images (png, jpg, gif, webp, etc.)
MediaCategory.DOCUMENT    # Documents (pdf, doc, docx, etc.)
MediaCategory.SPREADSHEET # Spreadsheets (xlsx, csv, etc.)
MediaCategory.CODE        # Code files
MediaCategory.AUDIO       # Audio files
MediaCategory.VIDEO       # Video files
MediaCategory.ARCHIVE     # Archives (zip, tar, etc.)
MediaCategory.NOTEBOOK    # Jupyter notebooks
MediaCategory.OTHER       # Other file types

Uploading Media

from foil import Foil

foil = Foil(
    api_key="your-api-key",
    base_url="https://api.foil.dev/api",
)

# Upload from file path
result = foil.upload_media("/path/to/image.png")
print(result["mediaId"])   # 'media_abc123'
print(result["category"])  # 'image'
print(result["filename"])  # 'image.png'
print(result["mimeType"])  # 'image/png'

# Upload from bytes
with open("/path/to/document.pdf", "rb") as f:
    content = f.read()
result = foil.upload_media(
    content,
    filename="document.pdf",
    mime_type="application/pdf",
)

# Upload with trace/span association
result = foil.upload_media(
    "/path/to/image.png",
    trace_id="trace_123",
    span_id="span_456",
    direction="input",  # or "output"
)

Content Blocks

Use content blocks to create multimodal input/output:

from foil import content, ContentBlock, MediaCategory

# Create multimodal content array
multimodal_input = content(
    "Please analyze this image:",
    ContentBlock.media(
        upload_result["mediaId"],
        category=MediaCategory.IMAGE,
        filename="photo.jpg",
        mime_type="image/jpeg",
    ),
    "Focus on the composition and colors."
)

Media Retrieval

# Get media information
media_info = foil.get_media(media_id)

# Get presigned URL for download
url_info = foil.get_media_url(media_id, "original")
print(url_info["url"])  # Presigned S3 URL

# Get extracted content URL (for documents)
extracted_url = foil.get_media_url(media_id, "extracted")

# Batch get multiple media
batch_info = foil.batch_media_info([media_id1, media_id2, media_id3])

Signals & Feedback

Record user feedback and custom signals for your traces:

from foil import Foil

foil = Foil(api_key="your-api-key")

# Record a signal for a specific trace
foil.record_signal({
    "traceId": "trace_123",
    "signalName": "user_satisfaction",
    "value": 4,
    "signalType": "feedback",
    "source": "user",
})

# Batch record signals
foil.record_signal_batch([
    {"traceId": "trace_123", "signalName": "thumbs", "value": True, "source": "user"},
    {"traceId": "trace_123", "signalName": "rating", "value": 5, "source": "user"},
])

# Get signals for a trace
signals = foil.get_trace_signals("trace_123")

OpenAI Integration

Automatic logging for OpenAI calls:

from openai import OpenAI
from foil import Foil

foil = Foil(api_key="your-foil-api-key")
client = foil.wrap_openai(OpenAI())

# All chat completion calls are now automatically logged
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello!"}]
)

Supports both streaming and non-streaming responses.

Experiments (A/B Testing)

Get variant assignments for experiments:

from foil import Foil

foil = Foil(api_key="your-api-key")

# Get assigned variant for a user
assignment = foil.get_experiment_variant("experiment_123", user_id)

if assignment["inExperiment"]:
    print(assignment["variantName"])  # 'control' or 'treatment'
    print(assignment["config"])       # {'model': 'gpt-4', 'temperature': 0.7}

Low-Level API

For more control, use the Foil client directly:

from foil import Foil

foil = Foil(
    api_key="your-api-key",
    base_url="https://api.foil.dev/api",
)

# Manual span management
trace_id = foil.create_trace_id()
span_id = foil.create_span_id()

foil.start_span({
    "spanId": span_id,
    "traceId": trace_id,
    "name": "gpt-4",
    "agentName": "my-agent",
    "spanKind": "llm",
    "input": messages,
})

# ... do work ...

foil.end_span({
    "spanId": span_id,
    "traceId": trace_id,
    "agentName": "my-agent",
    "output": response,
    "tokens": {"prompt": 100, "completion": 50, "total": 150},
})

# Get trace data
trace = foil.get_trace(trace_id)

# List traces
traces = foil.list_traces(
    agent_name="my-agent",
    limit=10,
    from_date="2024-01-01T00:00:00Z",
)

API Reference

Foil Client

Initialization

foil = Foil(api_key="your-api-key", base_url="https://api.getfoil.ai/api")

Parameter	Type	Required	Description
api_key	str	Yes	Your Foil API key
base_url	str	No	API base URL (default: https://api.getfoil.ai)

Semantic Search Methods

Method	Description
semantic_search(query, **options)	Search spans using natural language
find_similar_traces(trace_id, **options)	Find traces similar to a given trace
get_semantic_search_status(agent_id=None)	Get embedding statistics

Custom Evaluation Methods

Method	Description
get_evaluation_templates()	List available evaluation templates
get_agent_evaluations(agent_id)	List evaluations for an agent
get_evaluation(agent_id, evaluation_id)	Get evaluation details
create_evaluation(agent_id, data)	Create a custom evaluation
update_evaluation(agent_id, evaluation_id, data)	Update an evaluation
delete_evaluation(agent_id, evaluation_id)	Delete an evaluation
test_evaluation(agent_id, evaluation_id, data)	Test with sample data
clone_evaluation_template(agent_id, template_id)	Clone a template
add_evaluation_example(agent_id, evaluation_id, data)	Add few-shot example
remove_evaluation_example(agent_id, evaluation_id, example_id)	Remove example
get_evaluation_analytics(agent_id, evaluation_id)	Get evaluation metrics

Media Methods

Method	Description
upload_media(file, **options)	Upload media for multimodal content
get_media(media_id, content=None)	Get media information
get_media_url(media_id, content="original")	Get presigned download URL
batch_media_info(media_ids)	Get info for multiple media

Signal Methods

Method	Description
record_signal(data)	Record a signal
record_signal_batch(signals)	Record multiple signals
get_trace_signals(trace_id)	Get signals for a trace

Trace Methods

Method	Description
start_span(data)	Start a span
end_span(data)	End a span
get_trace(trace_id)	Get trace with all spans
list_traces(**options)	List traces with filters

Experiment Methods

Method	Description
get_experiment_variant(experiment_id, identifier)	Get variant assignment

Legacy Methods (Deprecated)

Method	Replacement
start_invocation(data)	Use start_span(data)
end_invocation(data)	Use end_span(data)

Examples

See the examples/ directory for complete working examples:

• semantic_search_example.py - Semantic search usage
• custom_evaluations_example.py - Custom evaluations

Run examples:

export FOIL_API_KEY=your-api-key
export FOIL_BASE_URL=https://api.getfoil.ai/api
export FOIL_AGENT_ID=your-agent-id

python examples/semantic_search_example.py
python examples/custom_evaluations_example.py

License

MIT