alignmenter 0.2.0

Persona-aligned evaluation for conversational AI

📚 Documentation • Quickstart • Quick Start Guide • CLI Reference • License

Overview

Alignmenter is a lightweight, production-ready evaluation toolkit for auditing conversational AI systems across three core dimensions:

• 🎨 Authenticity: Does the assistant stay on-brand?
• 🛡️ Safety: Does it avoid harmful or policy-violating outputs?
• ⚖️ Stability: Are responses consistent across sessions?

Unlike generic LLM evaluation frameworks, Alignmenter is purpose-built for persona alignment—ensuring your AI assistant speaks with your unique voice while staying safe and stable.

Quickstart

Installation

# Clone the repository
git clone https://github.com/justinGrosvenor/alignmenter.git
cd alignmenter

# Create virtual environment
python -m venv env
source env/bin/activate  # On Windows: env\Scripts\activate

# Install with all optional extras for development
pip install -e "./alignmenter[dev]"

Install from PyPI

The core install is lightweight — no torch, no scikit-learn. It runs the default scoring path (hashed embeddings + keyword safety, optionally blended with an LLM judge) out of the box:

pip install alignmenter
alignmenter init
alignmenter run --config configs/run.yaml

Add optional extras only when you need them:

# Example: better local embeddings + offline safety classifier
pip install "alignmenter[ml]"
alignmenter run --config configs/run.yaml --embedding sentence-transformer:all-MiniLM-L6-v2

Offline safety classifier: [ml] includes transformers for the offline classifier (ProtectAI/distilled-safety-roberta). Without it, Alignmenter falls back to a lightweight heuristic classifier. See docs/offline_safety.md for details.

Run Your First Evaluation

# Set API key (for embedding and judge models)
export OPENAI_API_KEY="your-key-here"

# Run evaluation on demo dataset (regenerates transcripts via the provider)
alignmenter run \
  --model openai:gpt-4o-mini \
  --dataset datasets/demo_conversations.jsonl \
  --persona configs/persona/default.yaml

# Reuse existing transcripts without hitting the provider
alignmenter run --config configs/run.yaml

# View interactive HTML report
alignmenter report --last

# Sanitize a dataset in-place or to a new file
alignmenter dataset sanitize datasets/demo_conversations.jsonl --out datasets/demo_sanitized.jsonl

# Generate fresh transcripts (requires provider access)
alignmenter run --config configs/run.yaml --generate-transcripts

Output:

Loading dataset: 60 turns across 10 sessions
✓ Brand voice score: 0.82 (range: 0.78-0.86)
✓ Safety score: 0.97
✓ Consistency score: 0.94
Report written to: reports/demo/2025-11-03T00-14-01_alignmenter_run/index.html

Features

🎯 Three-Dimensional Scoring

Authenticity

• Embedding similarity: Measures semantic alignment with persona examples
• Trait model: Logistic regression on linguistic features (trained via calibration)
• Lexicon matching: Enforces preferred/avoided vocabulary
• Bootstrap CI: Statistical confidence intervals for reliability

Safety

• Keyword classifier: Fast pattern matching for common violations
• LLM judge: GPT-4 as a safety oracle with budget controls
• Offline classifier: ProtectAI's distilled-safety-roberta (no API calls)
• Fused scoring: Weighted ensemble of rule-based + model-based signals
• Adversarial testing: Built-in safety traps in demo datasets

Stability

• Cosine variance: Detects semantic drift across conversation turns
• Session clustering: Identifies divergent response patterns
• Temporal analysis: Tracks consistency over time

📊 Rich Reporting

• Interactive HTML: Grade-based report cards with charts (Chart.js)
• JSON export: Machine-readable results for CI/CD pipelines
• CSV downloads: Per-metric exports for spreadsheet analysis
• Turn-level explorer: Drill down into individual responses

🔧 Production-Ready

• Multi-provider support: OpenAI, Anthropic, local (vLLM, Ollama)
• Budget guardrails: Halt runs at 90% of judge API budget
• Cost projection: Estimate expenses before execution
• Reproducibility: Logs Python version, model, seed, timestamps
• PII sanitization: Built-in scrubbing for production data

🚀 Developer Experience

• CLI-first: Simple commands for evaluation, calibration, reporting
• YAML configuration: Declarative persona packs and run configs
• Python API: Programmatic access for custom workflows
• Comprehensive tests: 69+ unit tests with pytest
• Type safety: Full type hints throughout

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        Alignmenter CLI                          │
│  alignmenter run / report / calibrate / bootstrap / sanitize    │
└─────────────────────────────────────────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────────┐
│                          Runner                                 │
│  Orchestrates evaluation: load data → score → report            │
└─────────────────────────────────────────────────────────────────┘
                               │
           ┌───────────────────┼───────────────────┐
           ▼                   ▼                   ▼
   ┌──────────────┐    ┌──────────────┐    ┌──────────────┐
   │ Authenticity │    │    Safety    │    │  Stability   │
   │    Scorer    │    │    Scorer    │    │    Scorer    │
   └──────────────┘    └──────────────┘    └──────────────┘
           │                   │                   │
           │                   │                   │
   ┌──────────────┐    ┌──────────────┐    ┌──────────────┐
   │  Embeddings  │    │  LLM Judge   │    │   Cosine     │
   │  Trait Model │    │  Keywords    │    │  Variance    │
   │   Lexicon    │    │  Fusion      │    │  Clustering  │
   └──────────────┘    └──────────────┘    └──────────────┘
                               │
                               ▼
           ┌───────────────────────────────────────┐
           │         Reporting Layer               │
           │  HTML / JSON / CSV / Interactive UI   │
           └───────────────────────────────────────┘

Key Components

📚 Documentation

Full documentation available at docs.alignmenter.com

Quick links:

• Quick Start Guide - Get started in 5 minutes
• Installation - Install and setup
• CLI Reference - Complete command reference
• Persona Guide - Configure your brand voice
• Calibration Guide - Advanced calibration workflow
• Safety Guide - Offline safety classifier
• LLM Judges - Qualitative analysis
• Contributing - How to contribute

Case Studies

• Wendy's Twitter Voice - End-to-end calibration example using the included case-study assets. (Available when running from the source repo; not included in the PyPI wheel.)

Usage Examples

Evaluate Multiple Models

# Compare GPT-4 vs Claude
alignmenter run \
  --model openai:gpt-4 \
  --compare anthropic:claude-3-5-sonnet-20241022 \
  --dataset datasets/demo_conversations.jsonl \
  --persona configs/persona/default.yaml

Custom Judge and Embeddings

# Use Claude as safety judge, local embeddings
alignmenter run \
  --model openai:gpt-4o-mini \
  --judge anthropic:claude-3-5-sonnet-20241022 \
  --embedding sentence-transformer:all-MiniLM-L6-v2 \
  --dataset datasets/demo_conversations.jsonl \
  --persona configs/persona/default.yaml

Bootstrap Synthetic Dataset

# Generate 50 conversations with adversarial traps
alignmenter bootstrap-dataset \
  --out datasets/my_test.jsonl \
  --sessions 50 \
  --safety-trap-ratio 0.15 \
  --brand-trap-ratio 0.20 \
  --seed 42

Calibrate Persona Traits

# Train trait model from labeled data
alignmenter calibrate-persona \
  --persona-path configs/persona/mybot.yaml \
  --dataset annotations.jsonl \
  --out configs/persona/mybot.traits.json \
  --epochs 300

Sanitize Production Data

# Remove PII before evaluation
alignmenter dataset sanitize prod_logs.jsonl \
  --out datasets/sanitized.jsonl \
  --no-use-hashing

Persona Configuration

Define your brand voice in YAML:

# configs/persona/mybot.yaml
id: mybot
name: "MyBot Assistant"
description: "Professional, evidence-driven, technical"

voice:
  tone: ["professional", "precise", "measured"]
  formality: "business_casual"

  # Preferred vocabulary
  lexicon:
    preferred:
      - "baseline"
      - "signal"
      - "alignment"
      - "evidence-based"
    avoided:
      - "lol"
      - "bro"
      - "hype"
      - "vibes"

# Example on-brand responses (for embedding similarity)
examples:
  - "Our baseline analysis indicates a 15% improvement in alignment metrics."
  - "The signal-to-noise ratio suggests this approach is viable."
  - "Let's establish a clear baseline before proceeding."

# Trait model weights (generated by calibration)
traits:
  weights: [0.12, -0.34, 0.08, ...]  # Learned from annotations
  vocabulary: ["baseline", "signal", ...]

API Usage

Runner coordinates transcript preparation, scoring, and report generation. It takes a RunConfig plus a list of scorers, and execute() returns the path to the timestamped report directory (JSON + HTML are written for you).

import json
from pathlib import Path

from alignmenter.runner import RunConfig, Runner
from alignmenter.scorers.authenticity import AuthenticityScorer
from alignmenter.scorers.safety import SafetyScorer
from alignmenter.scorers.stability import StabilityScorer

config = RunConfig(
    model="openai:gpt-4o-mini",
    dataset_path=Path("datasets/demo_conversations.jsonl"),
    persona_path=Path("configs/persona/default.yaml"),
)

# Pass a judge to AuthenticityScorer/SafetyScorer to blend LLM judgment in;
# omit it (as here) for a fully offline, deterministic run.
scorers = [
    AuthenticityScorer(persona_path=config.persona_path, embedding="hashed"),
    SafetyScorer(keyword_path=Path("configs/safety_keywords.yaml")),
    StabilityScorer(embedding="hashed"),
]

# generate_transcripts=False reuses recorded transcripts (no provider calls).
runner = Runner(config, scorers, generate_transcripts=False)
run_dir = runner.execute()  # -> Path to reports/<timestamp>_<run_id>/

results = json.loads((run_dir / "results.json").read_text())
primary = results["scores"]["primary"]
auth = primary["authenticity"]
print(f"Authenticity: {auth['mean']:.3f}  (basis: {auth['basis']})")
print(f"Safety:       {primary['safety']['score']:.3f}")
print(f"Stability:    {primary['stability']['stability']:.3f}")

CI/CD Integration

# .github/workflows/eval.yml
name: Persona Evaluation

on: [push, pull_request]

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install Alignmenter
        run: pip install alignmenter

      - name: Run Evaluation
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          alignmenter run \
            --model openai:gpt-4o-mini \
            --dataset datasets/ci_test.jsonl \
            --persona configs/persona/default.yaml \
            --judge-budget 100

      - name: Upload Report
        uses: actions/upload-artifact@v3
        with:
          name: evaluation-report
          path: reports/

Development

Running Tests

# All tests
pytest

# With coverage
pytest --cov=src/alignmenter --cov-report=html

# Specific test file
pytest tests/test_scorers.py -v

Code Quality

# Type checking
mypy src/

# Linting
ruff check src/

# Formatting
black src/ tests/

Local Development

# Install in editable mode with dev dependencies
pip install -e .[dev]

# Run from source
python -m alignmenter.cli run --help

# Generate report from last run
make report-last

Roadmap

Completed ✅

• Three-dimensional scoring (authenticity, safety, stability)
• Multi-provider support (OpenAI, Anthropic, local models)
• HTML report cards with interactive charts
• Offline safety classifier (distilled-safety-roberta)
• LLM judges for qualitative analysis
• Budget guardrails and cost tracking
• PII sanitization tools
• Calibration workflow and diagnostics

In Progress 🚧

• Multi-language support (non-English personas)
• Batch processing optimizations
• Additional embedding providers

Future Considerations 💭

• Synthetic test case generation
• Custom metric plugins
• Advanced trait models (neural networks)

Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

Areas we'd love help with:

• Additional persona packs (different brand voices)
• Language support beyond English
• Integration with other LLM providers
• Performance optimizations for large datasets

License

Apache License 2.0 - see LICENSE for details.

Citation

If you use Alignmenter in research, please cite:

@software{alignmenter2024,
  title={Alignmenter: A Framework for Persona-Aligned Conversational AI Evaluation},
  author={Alignmenter Contributors},
  year={2025},
  url={https://github.com/justinGrosvenor/alignmenter},
  license={Apache-2.0}
}

Support

• Documentation: docs.alignmenter.com
• Issues: GitHub Issues
• Discussions: GitHub Discussions

Made with ❤️ by the Alignmenter team