spyglasses 1.0.2

AI Agent Detection and Management for Python web applications

Spyglasses Python SDK

AI Agent Detection and Management for Python web applications.

The Spyglasses Python SDK enables you to detect AI Agents, bots, crawlers, and referrers in your Python web applications. It provides comprehensive AI SEO, shows you when your site features in ChatGPT, Claude, Perplexity, and other AI assistant chat platforms. It can also prevent your site's content from being used for training AI by blocking the crawlers that scrape your content for training. It integrates seamlessly with popular Python web frameworks including Django, Flask, and FastAPI.

Features

• AI Bot Detection: Identify AI crawlers, model trainers, and assistants
• AI Referrer Detection: Track traffic from AI platforms like ChatGPT, Claude, Perplexity
• Framework Integration: Built-in middleware for Django, Flask, FastAPI, WSGI, and ASGI
• Pattern Management: Auto-sync detection patterns from Spyglasses API
• Custom Rules: Configure custom blocking and allowing rules
• Request Logging: Automatic logging to Spyglasses collector for analytics
• Thread Safe: Designed for multi-threaded web applications
• Type Hints: Full typing support for better development experience

Installation

Basic Installation

pip install spyglasses

Framework-Specific Installation

# For Django projects
pip install 'spyglasses[django]'

# For Flask projects  
pip install 'spyglasses[flask]'

# For FastAPI projects
pip install 'spyglasses[fastapi]'

# For development (includes testing tools)
pip install 'spyglasses[dev]'

Quick Start

1. Get Your API Key

Sign up at spyglasses.io and get your API key from the dashboard.

2. Set Environment Variables

export SPYGLASSES_API_KEY="your-api-key-here"
export SPYGLASSES_DEBUG="true"  # Optional: for debugging

3. Framework Integration

Django

Add to your settings.py:

MIDDLEWARE = [
    'spyglasses.middleware.DjangoMiddleware',
    # ... your other middleware
]

# Optional: Configure Spyglasses settings
SPYGLASSES_API_KEY = os.getenv('SPYGLASSES_API_KEY')
SPYGLASSES_DEBUG = True

Or configure programmatically:

# In your middleware configuration
from spyglasses.middleware import DjangoMiddleware

# Add to Django middleware
MIDDLEWARE = [
    DjangoMiddleware,
    # ... other middleware
]

Flask

from flask import Flask
from spyglasses.middleware import FlaskMiddleware

app = Flask(__name__)

# Initialize middleware
middleware = FlaskMiddleware(
    app=app,
    api_key="your-api-key",
    debug=True
)

# Or use as decorator for specific routes
from spyglasses.middleware.flask import spyglasses_middleware

@app.route('/api/data')
@spyglasses_middleware(api_key="your-api-key")
def get_data():
    return {"data": "sensitive information"}

FastAPI

from fastapi import FastAPI
from spyglasses.middleware.fastapi import setup_spyglasses

app = FastAPI()

# Setup middleware
setup_spyglasses(
    app,
    api_key="your-api-key",
    debug=True
)

# Alternative manual setup
from spyglasses.middleware import FastAPIMiddleware
app.add_middleware(
    FastAPIMiddleware,
    api_key="your-api-key",
    debug=True
)

WSGI Applications

from spyglasses.middleware import WSGIMiddleware

def application(environ, start_response):
    # Your WSGI app
    pass

# Wrap with Spyglasses middleware
application = WSGIMiddleware(
    application,
    api_key="your-api-key",
    debug=True
)

ASGI Applications

from spyglasses.middleware import ASGIMiddleware

async def application(scope, receive, send):
    # Your ASGI app
    pass

# Wrap with Spyglasses middleware
application = ASGIMiddleware(
    application,
    api_key="your-api-key",
    debug=True
)

Configuration

Environment Variables

Variable	Description	Default
SPYGLASSES_API_KEY	Your Spyglasses API key	Required
SPYGLASSES_DEBUG	Enable debug logging	false
SPYGLASSES_AUTO_SYNC	Auto-sync patterns on startup	true
SPYGLASSES_COLLECT_ENDPOINT	Custom collector endpoint	https://www.spyglasses.io/api/collect
SPYGLASSES_PATTERNS_ENDPOINT	Custom patterns endpoint	https://www.spyglasses.io/api/patterns
SPYGLASSES_CACHE_TTL	Pattern cache TTL in seconds	86400 (24 hours)
SPYGLASSES_PLATFORM_TYPE	Platform identifier	python

Programmatic Configuration

from spyglasses import Configuration, Client

# Create custom configuration
config = Configuration(
    api_key="your-api-key",
    debug=True,
    auto_sync=True,
    platform_type="my-app",
    exclude_paths=["/admin", "/internal"]
)

# Use with client
client = Client(config)

# Or configure globally
import spyglasses
spyglasses.configure(config)

Direct Usage (Without Middleware)

from spyglasses import Client

# Initialize client
client = Client()

# Detect bot
user_agent = "Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)"
result = client.detect(user_agent)

if result.is_bot:
    print(f"Bot detected: {result.info.company} - {result.info.type}")
    if result.should_block:
        print("This bot should be blocked")

# Detect AI referrer
referrer = "https://chat.openai.com/"
result = client.detect_ai_referrer(referrer)

if result.source_type == "ai_referrer":
    print(f"AI referrer detected: {result.info.name}")

# Combined detection
result = client.detect(user_agent, referrer)

# Log request manually
request_info = {
    "url": "https://example.com/api/data",
    "user_agent": user_agent,
    "ip_address": "192.168.1.1",
    "request_method": "GET",
    "request_path": "/api/data",
    "request_query": "",
    "referrer": referrer,
    "response_status": 200,
    "response_time_ms": 150,
    "headers": {"Host": "example.com"}
}

client.log_request(result, request_info)

Advanced Usage

Custom Exclusions

from spyglasses.middleware import DjangoMiddleware

# Exclude additional paths and file extensions
middleware = DjangoMiddleware(
    exclude_paths=["/health", "/metrics", "/admin"],
    exclude_extensions=[".ico", ".png", ".css"]
)

Pattern Synchronization

from spyglasses import Client

client = Client()

# Manual pattern sync
result = client.sync_patterns()
if isinstance(result, str):
    print(f"Sync failed: {result}")
else:
    print(f"Synced {len(result.patterns)} patterns")

# Check sync status
print(f"Last sync: {client.last_pattern_sync}")
print(f"Pattern version: {client.pattern_version}")

Middleware Options

All middleware classes support these options:

middleware = FrameworkMiddleware(
    api_key="your-api-key",
    debug=True,
    collect_endpoint="https://custom.endpoint.com/collect",
    patterns_endpoint="https://custom.endpoint.com/patterns", 
    auto_sync=True,
    platform_type="my-platform",
    cache_ttl=3600,  # 1 hour
    exclude_paths=["/admin", "/internal"],
    exclude_extensions=[".ico", ".png"]
)

Bot Categories

Spyglasses detects various types of bots:

AI Assistants

• ChatGPT-User (OpenAI user queries)
• Claude-User (Anthropic user queries)
• Perplexity-User (Perplexity user queries)
• Gemini-User (Google user queries)

AI Model Training Crawlers

• GPTBot (OpenAI training crawler)
• ClaudeBot (Anthropic training crawler)
• CCBot (Common Crawl)
• meta-externalagent (Meta training crawler)
• Applebot-Extended (Apple training crawler)

AI Referrers

• Traffic from ChatGPT, Claude, Perplexity, Gemini, Microsoft Copilot

Error Handling

The SDK is designed to be non-intrusive. If there are issues with the Spyglasses service:

• Pattern sync failures fall back to default patterns
• Network errors don't block requests
• Invalid configurations log warnings but don't crash
• All operations are wrapped in try/catch blocks

Development

Setting up Development Environment

git clone https://github.com/spyglasses/spyglasses-python.git
cd spyglasses-python

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install in development mode
pip install -e ".[dev]"

# Run tests
pytest

# Run type checking
mypy src/

# Format code
black src/ tests/
ruff src/ tests/

Running Tests

# Run all tests
pytest

# Run with coverage
pytest --cov=spyglasses

# Run specific test file
pytest tests/test_client.py

# Run integration tests
pytest -m integration

License

MIT License. See LICENSE for details.

Support

• Documentation: spyglasses.io/docs
• Issues: GitHub Issues
• Support: [e-mail]

Changelog

See CHANGELOG.md for version history and changes.