Automatic Goggles

Post-Call Analysis & Conversational Evaluation

Extract structured fields from transcripts with confidence scores and evaluate conversation quality using Assertive LLM-as-a-Judge

Automatic-Goggles? | Installation | Quick Start | Features | Use-Cases

🎯 What is Automatic Goggles?

Automatic Goggles is a lightweight, production-ready Python package for analyzing conversation transcripts. It provides two core capabilities:

1. 🔍 Field Extraction

Extract structured data (names, emails, dates, custom fields) from transcripts with confidence scores

2. ⚖️ Conversation Evaluation

Assess conversation quality against custom assertions using LLM-as-a-Judge

Built for voice agent post-call analysis, customer support quality assurance, and conversational AI evaluation.

🚀 Installation

pip install automatic-goggles

Requires Python 3.8+

🚀 Examples

Field Extraction in 30 Seconds

from transtype import TranscriptProcessor

fields = [
    {
        "field_name": "agent_email",
        "field_type": "string",
        "format_example": "agent@company.com",
        "field_description": "The agent's email address for follow-up communication"
    },
    {
        "field_name": "agent_name",
        "field_type": "string",
        "format_example": "John Doe",
        "field_description": "Full name of the customer service agent"
    }
]

processor = TranscriptProcessor(
    api_key="your-openai-api-key",
    fields=fields,
    include_reasoning=True
)

conversation = {
    "messages": [
        {"role": "assistant", "content": "My name is Sarah Chen, you can reach me at sarah@example.com"},
        {"role": "user", "content": "Thanks, I'll email you"}
    ]
}

result = processor.process(conversation)

Output:

{
    "fields": [
        {
            "field_name": "agent_email",
            "field_value": "sarah@example.com",
            "field_confidence": 0.92,
            "field_reason": "Email explicitly mentioned by agent"
        },
        {
            "field_name": "agent_name",
            "field_value": "Sarah Chen",
            "field_confidence": 0.95,
            "field_reason": "Agent introduced herself by name"
        }
    ]
}

Conversation Evaluation in 30 Seconds

from transtype import AssertsEvaluator

evaluator = AssertsEvaluator(
    api_key="your-openai-api-key",
    evaluation_steps=[
        "Did the agent greet the customer politely?",
        "Did the agent ask clarifying questions?",
        "Did the agent resolve the customer's issue?",
        "Did the agent offer additional help?"
    ],
    threshold=0.7  # Pass threshold (0-1)
)

conversation = {
    "messages": [
        {"role": "user", "content": "My internet isn't working"},
        {"role": "assistant", "content": "Good morning! I'm sorry to hear that. When did this issue start?"},
        {"role": "user", "content": "This morning"},
        {"role": "assistant", "content": "Let me help you troubleshoot. Can you check if your router is powered on?"}
    ]
}

result = evaluator.evaluate(conversation)

Output:

{
    "result": {
        "score": 0.88,
        "success": true,
        "reason": "Agent demonstrated professionalism, asked clarifying questions, and initiated troubleshooting"
    }
}

🔥 Features

Field Extraction

• ✅ Confidence-Weighted Scoring - Log probability-based confidence scores (0-1)
• ✅ Contextual Descriptions - Improve accuracy with detailed field descriptions
• ✅ Flexible Reasoning - Toggle explanations on/off for performance/cost optimization
• ✅ Multi-Field Support - Extract multiple fields in one pass
• ✅ Format Examples - Guide extraction with format examples

Conversation Evaluation

• ✅ LLM-as-a-Judge - Research-backed evaluation using GPT models
• ✅ Custom Assertions - Define your own quality criteria
• ✅ Weighted Scoring - Confidence-weighted scores
• ✅ Pass/Fail Thresholds - Configurable success criteria
• ✅ Multi-Turn Support - Evaluate entire conversations

Technical Highlights

• ⚡ Fast & Cost-Effective - Optional reasoning for performance tuning
• 🧪 Production-Ready - Confidence scores for reliability filtering
• 📊 Transparent - Get reasoning explanations for every extraction/evaluation

💡 Use Cases

🛠️ Field Definition Schema

Each field requires:

Example:

{
    "field_name": "customer_phone",
    "field_type": "string",
    "format_example": "(555) 123-4567",
    "field_description": "The customer's primary phone number for callbacks. Look for 10-digit numbers in various formats."
}

🤝 Contributing

We welcome contributions! Here's how you can help:

1. Report Bugs - Open an issue on GitHub
2. Feature Requests - Suggest new features via issues
3. Pull Requests - Submit PRs for bug fixes or features
4. Documentation - Improve docs or add examples

📝 License

MIT License - see LICENSE for details

🙏 Acknowledgments

• Inspired by DeepEval's ConversationalGEval for confidence-weighted scoring
• Built with DSPy for robust LLM orchestration
• Powered by OpenAI language models

📬 Contact & Support

• GitHub Issues: Report bugs or request features
• Email: ashishorkalra@gmail.com
• Package: PyPI - automatic-goggles