llmq-gate 0.1.1

LLM Quality Gate - A provider-agnostic evaluation framework for LLM applications

LLMQ

Regression Testing & Quality Gates for LLM Applications

Open-source framework to catch silent LLM failures before they reach production.

Website · Quick Start · PyPI · Dashboard Demo · Contributing

---

The Problem

LLM applications fail silently. There's no stack trace when your summarizer starts hallucinating. No exception when your classifier drifts. A prompt change that works in development can quietly degrade production. Model updates break existing functionality overnight.

Without systematic testing, these regressions go undetected until users complain.

Common regressions LLMQ catches:

• Prompt optimization improves one task but degrades another
• Model updates change response formats, breaking downstream parsing
• Provider API changes affect response quality
• Temperature adjustments reduce output consistency
• Context length changes truncate important information

Quick Start

Get running in under 5 minutes:

# Install
pip install llmq-gate

# Initialize project
llmq init

# Set your API key
echo "GROQ_API_KEY=your_key_here" >> .env

# Run your first evaluation
llmq eval --provider groq

View results in the browser:

llmq dashboard
# → http://localhost:8000

How It Works

Dataset → LLM Provider → Metrics Engine → Quality Gates → Pass / Fail

1. Define test cases in evals/dataset.json with inputs, expected outputs, and context
2. Run evaluations against any supported provider
3. Metrics are computed automatically — task success, relevance, hallucination, consistency
4. Quality gates pass or fail based on your configured thresholds
5. Results are stored for historical tracking and comparison

Supported Providers

Provider	Models	API Key	Cost
Groq	Llama 3.1, Mixtral	Required	Free tier
OpenAI	GPT-3.5, GPT-4	Required	Paid
Claude	Claude 3 Haiku / Sonnet	Required	Paid
Gemini	Gemini 1.5 Flash / Pro	Required	Free tier
HuggingFace	Open models	Required	Free
OpenRouter	100+ models	Required	Varies
Ollama	Local models	—	Free
LocalAI	Local models	—	Free

CI/CD Integration

Add quality gates to your pull request workflow. Builds fail automatically when LLM performance drops below your thresholds.

# .github/workflows/llm-quality-gate.yml
name: LLM Quality Gate

on:
  pull_request:
    paths: ['prompts/**', 'llm/**', 'llmq.yaml']

jobs:
  quality-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install LLMQ
        run: pip install llmq-gate

      - name: Run Quality Gate
        env:
          GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
        run: llmq eval --provider groq --fail-on-gate

Metrics

Metric	Method	Description
Task Success	Exact match + semantic similarity	Did the model produce the correct answer?
Relevance	Embedding-based cosine similarity	Is the response relevant to the input?
Hallucination	LLM-as-judge detection	Did the model fabricate information?
Consistency	Multi-run variance analysis	Are responses stable across runs?

Dashboard

llmq dashboard

The interactive web dashboard provides historical performance tracking, provider comparison charts, quality gate pass/fail trends, and test case drill-down analysis.

🎬 Watch the full CLI + Dashboard walkthrough →

v0.1.1 Highlights

• Unified configuration filename: llmq.yaml everywhere.
• Config auto-discovery from current directory upward (similar to pyproject.toml lookup).
• llmq eval now supports standalone mode by falling back to local engine if dashboard API is unavailable.
• CLI exit codes are standardized:
  • 0: quality gate passed
  • 1: quality gate failed or runtime error
  • 2: configuration error (e.g., missing llmq.yaml)

Configuration

llmq.yaml — project-level settings:

llm:
  default_provider: "groq"
  temperature: 0.0
  max_tokens: 1000

providers:
  groq:
    api_key_env: "GROQ_API_KEY"
    model: "llama-3.1-8b-instant"
  openai:
    api_key_env: "OPENAI_API_KEY"
    model: "gpt-3.5-turbo"

quality_gates:
  task_success_threshold: 0.8
  relevance_threshold: 0.7
  hallucination_threshold: 0.1

evals/dataset.json — test cases:

{
  "test_cases": [
    {
      "id": "example_1",
      "task_type": "question_answering",
      "input": "What is the capital of France?",
      "expected_output": "Paris",
      "context": "Geography question",
      "reference": "Paris is the capital of France."
    }
  ]
}

CLI Reference

# Setup
llmq init                                    # Initialize new project
llmq doctor                                  # Check system health

# Evaluation
llmq eval --provider groq                    # Run evaluation
llmq eval --provider openai --fail-on-gate   # CI mode (exit 1 on gate failure)
llmq compare                                 # Compare providers side-by-side

# Management
llmq providers                               # List provider status
llmq runs --limit 10                         # View recent runs
llmq dashboard                               # Start web dashboard
llmq settings --set '{"quality_gates": {"task_success_threshold": 0.9}}'

Migration Guide (<=0.1.0 -> 0.1.1)

1. Rename existing config.yaml to llmq.yaml.
2. Update scripts to use --config-path (or continue using --config) when you need an explicit location.
3. Remove hard dependency on llmq dashboard for CLI evaluations; llmq eval now runs standalone if API is unavailable.
4. If you parse CLI statuses in CI, adopt the documented exit codes (0/1/2).

API

# Start an evaluation
curl -X POST http://localhost:8000/api/v1/evaluate \
  -H "Content-Type: application/json" \
  -d '{"provider": "groq"}'

# Get run history
curl http://localhost:8000/api/v1/runs

# Compare providers
curl http://localhost:8000/api/v1/compare

Contributing

Contributions are welcome — whether it's a bug fix, new provider integration, docs improvement, or feature request.

git clone https://github.com/Emart29/llm-quality-gate.git
cd llm-quality-gate
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install -e .
llmq doctor               # Verify setup

1. Fork the repository
2. Create a feature branch: git checkout -b feature-name
3. Make changes and add tests
4. Run tests: python -m pytest tests/ -v
5. Submit a pull request

Roadmap

v1.1 — Custom metric plugins · Slack/Discord webhooks · A/B testing framework · Performance benchmarking

v1.2 — Multi-language datasets · Advanced regression analysis · Cost tracking per provider · Distributed evaluation

v2.0 — Visual prompt debugging · Automated prompt optimization · Enterprise SSO · Advanced analytics

License

MIT — see LICENSE for details.

---

⭐ Star on GitHub · 📦 PyPI · 🌐 Website