pixie-qa 0.5.0

Automated quality assurance for AI applications

pixie-qa

An agent skill that makes coding agents the QA engineer for LLM applications.

What the Skill Does

The qa-eval skill guides your coding agent through the full eval-based QA loop for LLM applications:

1. Understand the code — read the codebase, trace the data flow, learn what the code is supposed to do
2. Instrument it — use wrap() for data-object tracing and OpenInference auto-instrumentation for LLM span capture
3. Build a dataset — create JSON datasets of representative inputs and expected outputs
4. Write eval tests — generate test_*.py files with appropriate evaluators
5. Run the tests — pixie test to run all evals and report per-case scores
6. Analyse results — pixie analyze <test_id> to get LLM-generated analysis of test results
7. Investigate failures — diagnose failures, fix, repeat

Getting Started

1. Add the skill to your coding agent

npx skills add yiouli/pixie-qa

The accompanying python package would be installed by the skill automatically when it's used.

2. Ask coding agent to set up evals

Open a conversation and say something like when developing a python based AI project:

"setup QA for my agent"

Your coding agent will read your code, instrument it, build a dataset from a few real runs, write and run eval-based tests, investigate failures and fix.

Python Package

The pixie-qa Python package (imported as pixie) is what Claude installs and uses inside your project. API docs are auto-generated by pdoc3 into docs/pixie/index.md via pre-commit. The markdown renderer uses scripts/pdoc_templates/text.mako so async functions and methods are explicitly shown as async def in signatures.

Install hooks once per clone:

uv run pre-commit install

Web UI

View all eval artifacts (results, markdown docs, datasets, and legacy scorecards) in a live-updating local web UI:

pixie start              # initializes pixie_qa/ (if needed) and opens http://localhost:7118
pixie start my_dir       # use a custom artifact root
pixie init               # scaffolds pixie_qa/ without starting the server

The web UI provides tabbed navigation for results, scorecards (legacy), datasets, and markdown files. Changes to artifacts are pushed to the browser in real time via SSE.

The server writes a server.lock file to the artifact root directory on startup (containing the port number) and removes it on shutdown, allowing other processes to discover whether the server is already running.

Configuration

Pixie reads configuration from environment variables and a local .env file through a single central config layer. Existing process env vars win over .env values.

Useful settings include:

• PIXIE_ROOT to move all generated artefacts under a different root directory
• PIXIE_RATE_LIMIT_ENABLED=true to enable evaluator throttling for pixie test
• PIXIE_RATE_LIMIT_RPS, PIXIE_RATE_LIMIT_RPM, PIXIE_RATE_LIMIT_TPS, and PIXIE_RATE_LIMIT_TPM to tune request and token throughput for LLM-as-judge evaluators