Declarative quality gate loops for AI-assisted development
Project description
pyqual
AI Cost Tracking
- ๐ค LLM usage: $1.0500 (7 commits)
- ๐ค Human dev: ~$500 (5.0h @ $100/h, 30min dedup)
Generated on 2026-03-29 using openrouter/qwen/qwen3-coder-next
Declarative quality gate loops for AI-assisted development.
One YAML file. One command. Pipeline iterates until your code meets quality thresholds.
pip install pyqual
pyqual init
pyqual run
The problem
You use Copilot, Claude, GPT. They generate code. But nobody checks if that code meets your quality standards before it hits code review. And nobody automatically iterates if it doesn't.
pyqual closes that gap: define metrics โ run tools โ check gates โ if fail, LLM fixes โ re-check โ repeat until pass.
How it works
pyqual.yaml defines everything:
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ metrics: โ
โ cc_max: 15 โ quality gates โ
โ vallm_pass_min: 90 โ
โ coverage_min: 80 โ
โ โ
โ stages: โ
โ - analyze (code2llm) โ
โ - validate (vallm) โ
โ - fix (llx/aider, when: fail) โ
โ - test (pytest) โ
โ โ
โ loop: โ
โ max_iterations: 3 โ
โ on_fail: report โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
pyqual run:
Iteration 1 โ analyze โ validate โ fix โ test โ check gates
โ
โโโ PASS โโโดโโ FAIL โโโ
โ โ
Done โ
Iteration 2...
pyqual.yaml
pipeline:
name: quality-loop
metrics:
cc_max: 15 # cyclomatic complexity per function
vallm_pass_min: 90 # vallm validation pass rate (%)
coverage_min: 80 # test coverage (%)
stages:
- name: analyze
run: code2llm ./ -f toon,evolution
- name: validate
run: vallm batch ./ --recursive --errors-json > .pyqual/errors.json
- name: fix
run: llx fix . --errors .pyqual/errors.json
when: metrics_fail # only runs if gates fail
timeout: 300 # seconds (optional)
optional: false # if true, failure is allowed
- name: test
run: pytest --cov --cov-report=json:.pyqual/coverage.json
when: always # always | metrics_fail | metrics_pass
loop:
max_iterations: 3
on_fail: report # report | create_ticket | block
Stage options
| Field | Type | Default | Description |
|---|---|---|---|
name |
string | required | Stage identifier |
run |
string | โ | Shell command to execute |
tool |
string | โ | Built-in tool preset (alternative to run) |
when |
string | always |
always, metrics_fail, metrics_pass |
timeout |
int | 0 | Seconds (0 = no limit) |
optional |
bool | false |
Allow failure without failing the pipeline |
Loop options
| Field | Type | Default | Description |
|---|---|---|---|
max_iterations |
int | 3 |
Maximum loop count |
on_fail |
string | report |
report โ print summary; create_ticket โ sync TODO.md via planfile; block โ exit non-zero immediately |
CLI
pyqual init
pyqual init [PATH]
Create pyqual.yaml with sensible defaults in PATH (default: current directory).
pyqual run
pyqual run [OPTIONS]
Options:
-c, --config PATH Config file [default: pyqual.yaml]
-w, --workdir PATH Working directory [default: .]
-n, --dry-run Preview pipeline without executing stages
-v, --verbose Show live pipeline log to stderr
Execute the full quality-gate loop. Exits non-zero if gates are not met.
pyqual run
pyqual run --config my-config.yaml --workdir ./src
pyqual run --dry-run # preview which stages would run
pyqual run --verbose # show stage output live
pyqual gates
pyqual gates [OPTIONS]
Options:
-c, --config PATH Config file [default: pyqual.yaml]
-w, --workdir PATH Working directory [default: .]
Check quality gates against currently collected metrics without running any stages. Useful after a manual tool run.
pyqual status
pyqual status [OPTIONS]
Options:
-c, --config PATH Config file [default: pyqual.yaml]
-w, --workdir PATH Working directory [default: .]
Show pipeline config summary and all metrics currently found in .pyqual/.
pyqual logs
pyqual logs [OPTIONS]
Options:
-w, --workdir PATH Working directory [default: .]
-n, --tail INT Show last N entries (0 = all) [default: 0]
-l, --level TEXT Filter by event type: stage_done, gate_check,
pipeline_start, pipeline_end
-f, --failed Show only failed stages/gates
-j, --json Raw JSON lines (ideal for LLM/llx consumption)
--sql TEXT Run arbitrary SQL against pipeline.db (advanced)
View structured pipeline logs from .pyqual/pipeline.db (written by nfo during every run).
pyqual logs # all entries
pyqual logs --tail 20 # last 20
pyqual logs --failed # only failures
pyqual logs --level gate_check # only gate results
pyqual logs --json --failed # JSON failures for LLM
pyqual logs --sql "SELECT * FROM pipeline_logs WHERE level='WARNING'"
pyqual bulk-init
pyqual bulk-init PATH [OPTIONS]
Options:
-n, --dry-run Preview without writing files
--no-llm Heuristic classification only (no LLM calls)
-m, --model TEXT Override LLM model for classification
--overwrite Regenerate even if pyqual.yaml already exists
--show-schema Print JSON schema used for LLM classification and exit
-j, --json Output results as JSON
Auto-generate pyqual.yaml for every subdirectory in a workspace. Detects project type (Python, Node.js, PHP, Rust, Go, shell, mixed) via LLM classification with JSON schema or heuristic fallback. Never overwrites unless --overwrite.
pyqual bulk-init /path/to/workspace
pyqual bulk-init /path/to/workspace --dry-run
pyqual bulk-init /path/to/workspace --no-llm
pyqual bulk-init /path/to/workspace --overwrite
pyqual bulk-init /path/to/workspace --show-schema
pyqual bulk-run
pyqual bulk-run PATH [OPTIONS]
Options:
-p, --parallel INT Max concurrent pyqual processes [default: 4]
-n, --dry-run Pass --dry-run to each project run
-t, --timeout INT Per-project timeout in seconds (0 = no limit) [default: 0]
-f, --filter TEXT Only run matching project names (repeatable)
--no-live Disable live dashboard, print final summary only
-v, --verbose Show last output line per project in dashboard
-j, --json Output final results as JSON
Run pyqual across all projects in a workspace with a real-time dashboard.
pyqual bulk-run /path/to/workspace
pyqual bulk-run /path/to/workspace --parallel 8
pyqual bulk-run /path/to/workspace --filter mylib --filter webapp
pyqual bulk-run /path/to/workspace --timeout 600
pyqual bulk-run /path/to/workspace --no-live # CI mode, no dashboard
pyqual bulk-run /path/to/workspace --json # JSON output
Live dashboard:
pyqual bulk-run running:3 pass:12 fail:1 err:0 queue:43 total:59
โโโโโโโโโโโโโโโโณโโโโโโโโโโโโโณโโโโโโโโณโโโโโโโโโโโณโโโโโโโโโโโโโณโโโโโโโโณโโโโโโโ
โProject โ Status โ Iter โ Stage โ Progress โ Gates โ Time โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โaidesk โ โ
passed โ 2/3 โ โ 100% โ 2/2 โ12.3s โ
โallama โ ๐ running โ 1/3 โ validate โ โโโโโโโโโโ โ 0/2 โ 8.5s โ
โblog-pactown โ โ failed โ 3/3 โ โ 60% โ 1/2 โ45.2s โ
โโฆ โ โณ queued โ โ โ โ โ โ
โโโโโโโโโโโโโโโโดโโโโโโโโโโโโโดโโโโโโโโดโโโโโโโโโโโดโโโโโโโโโโโโโดโโโโโโโโดโโโโโโโ
pyqual mcp-fix
pyqual mcp-fix [OPTIONS]
Options:
-w, --workdir PATH Project directory on the host [default: .]
--project-path TEXT Path as seen by the MCP service container
--issues PATH Gate-failure JSON [default: .pyqual/errors.json]
--output PATH Report output path [default: .pyqual/llx_mcp.json]
--endpoint TEXT MCP SSE endpoint URL
--model TEXT Override the model selected by llx
--file TEXT File to focus on (repeatable)
--use-docker Run aider inside Docker
--docker-arg TEXT Extra Docker arguments (repeatable)
--task TEXT Analysis task hint for llx [default: quick_fix]
--json Print full JSON result
Run the llx-backed MCP fix workflow. Reads .pyqual/errors.json, sends issues to an MCP SSE service, and writes the result to .pyqual/llx_mcp.json.
Requires pip install pyqual[mcp]. Set PYQUAL_LLX_MCP_URL or pass --endpoint.
pyqual mcp-fix
pyqual mcp-fix --workdir . --project-path /workspace/project
pyqual mcp-fix --issues .pyqual/errors.json --model claude-3-5-sonnet
pyqual mcp-fix --file src/main.py --file src/utils.py
pyqual mcp-fix --json # machine-readable result
pyqual mcp-refactor
pyqual mcp-refactor [OPTIONS]
Same options as mcp-fix (except no --task). Runs the refactor workflow instead of the fix workflow.
pyqual mcp-service
pyqual mcp-service [OPTIONS]
Options:
--host TEXT Host interface to bind to [default: 0.0.0.0]
--port INT Port to listen on [default: 8000]
Start the persistent llx MCP SSE service with /health and /metrics endpoints. Requires pip install pyqual[mcp].
pyqual tickets
pyqual tickets todo [OPTIONS] # sync TODO.md via planfile markdown backend
pyqual tickets github [OPTIONS] # sync GitHub Issues via planfile GitHub backend
pyqual tickets all [OPTIONS] # sync both
Common options:
-w, --workdir PATH Repository root [default: .]
--dry-run Preview without changing files
--direction TEXT from | to | both [default: both]
Planfile-backed ticket management. When on_fail: create_ticket is set, pyqual automatically calls tickets todo after a failed run.
pyqual tickets todo --dry-run # preview
pyqual tickets github --direction to # push to GitHub
pyqual tickets all --direction from # pull from all backends
Requires planfile (included). For GitHub: set GITHUB_TOKEN.
See examples/ticket_workflow/ for a complete example.
pyqual plugin
pyqual plugin list [--tag TAG] # list available plugins
pyqual plugin search QUERY # search by name/description/tag
pyqual plugin info NAME # show details + YAML snippet
pyqual plugin add NAME [--workdir .] # append config to pyqual.yaml
pyqual plugin remove NAME [--workdir .] # remove config from pyqual.yaml
pyqual plugin validate [--workdir .] # check configured plugins
Manage built-in metric collector plugins.
pyqual plugin list
pyqual plugin list --tag security
pyqual plugin search llm
pyqual plugin info llx-mcp-fixer
pyqual plugin add security
pyqual plugin remove llm-bench
Built-in plugins:
| Name | Tags | Metrics produced |
|---|---|---|
llm-bench |
llm, benchmark | pass_at_1, code_bleu, ai_generated_pct |
hallucination |
llm, rag | faithfulness_score, hallucination_rate, prompt_token_efficiency |
sbom |
security, compliance | sbom_coverage, vuln_supply_chain |
i18n |
i18n, l10n | i18n_coverage, i18n_missing |
a11y |
accessibility | a11y_issues, a11y_critical |
repo-metrics |
git, health | bus_factor, commit_frequency, contributor_diversity |
security |
security, secrets | secrets_found, vuln_critical, vuln_total |
llx-mcp-fixer |
mcp, llx, fix | llx_fix_success, llx_fix_returncode, llx_tool_calls |
See examples/custom_plugins/ for building your own.
pyqual doctor
pyqual doctor
Check availability of all external tools (bandit, mypy, ruff, pylint, flake8, radon, interrogate, vulture, pytest, trufflehog, gitleaks, docker, code2llm, vallm, uvicorn). Prints install commands for missing tools.
pyqual tools
pyqual tools
List all built-in tool presets that can be used as tool: shortcuts in pipeline stages (e.g., tool: ruff, tool: pytest, tool: code2llm).
Python API
from pyqual import Pipeline, PyqualConfig
config = PyqualConfig.load("pyqual.yaml")
pipeline = Pipeline(config, workdir="./my-project")
result = pipeline.run()
if result.final_passed:
print(f"All gates passed in {result.iteration_count} iterations")
else:
print("Gates not met โ check result.iterations for details")
for it in result.iterations:
for gate in it.gates:
print(f" {gate}")
Check gates without running stages:
from pyqual import PyqualConfig, GateSet
config = PyqualConfig.load("pyqual.yaml")
results = GateSet(config.gates).check_all()
for r in results:
print("โ
" if r.passed else "โ", r)
See examples/basic/ for more API patterns.
LLM Integration
pyqual includes built-in LLM support via liteLLM. Configure via .env:
The convenience wrapper lives upstream in llx.llm; pyqual re-exports it so existing imports keep working.
OPENROUTER_API_KEY=sk-or-v1-...
LLM_MODEL=openrouter/qwen/qwen3-coder-next
from pyqual import get_llm
llm = get_llm()
response = llm.complete("Explain Python decorators")
print(response.content)
print(f"Cost: ${response.cost:.4f}")
See examples/llx/ for a full LLM-backed pipeline.
Docker-backed MCP fixer/refactor
The MCP client, service, workflow orchestration (LlxMcpRunResult, run_llx_fix_workflow, run_llx_refactor_workflow), issue parsing, and prompt building live in the upstream llx package (โฅ 0.1.47). pyqual re-exports them for backward compatibility. Install pyqual[mcp] to enable.
docker compose -f examples/llm_fix/docker-compose.yml up --build -d
pyqual plugin add llx-mcp-fixer
pyqual run
The plugin writes results to .pyqual/llx_mcp.json, gated via llx_fix_* metrics.
# Run workflows directly
pyqual mcp-fix --workdir . --project-path /workspace/project
pyqual mcp-refactor --workdir . --project-path /workspace/project
# Run standalone service
pyqual mcp-service --host 0.0.0.0 --port 8000
Set PYQUAL_LLX_MCP_URL to point clients at the service. See examples/llm_fix/ for a complete Docker Compose setup.
Metric sources
pyqual automatically collects from .toon files and .pyqual/ artifacts:
| Source | File | Metrics |
|---|---|---|
| Analysis | project/analysis.toon.yaml |
cc (CCฬ), critical |
| Validation | project/validation.toon.yaml |
vallm_pass |
.pyqual/errors.json |
error_count |
|
| Coverage | .pyqual/coverage.json |
coverage |
| Performance | .pyqual/asv.json |
bench_regression, bench_time |
.pyqual/mem.json |
mem_usage, cpu_time |
|
| Security | .pyqual/bandit.json |
bandit_high, bandit_medium, bandit_low |
.pyqual/secrets.json |
secrets_severity, secrets_count |
|
.pyqual/vulns.json |
vuln_critical, vuln_count |
|
.pyqual/sbom.json |
sbom_compliance, license_blacklist |
|
| Project health | .pyqual/vulture.json |
unused_count |
.pyqual/pyroma.json |
pyroma_score |
|
.pyqual/git_metrics.json |
git_branch_age, todo_count |
|
| LLM / AI | .pyqual/humaneval.json |
llm_pass_rate |
.pyqual/llm_analysis.json |
llm_cc, hallucination_rate, prompt_bias_score, agent_efficiency |
|
.pyqual/llx_mcp.json |
llx_fix_success, llx_fix_returncode, llx_tool_calls, llx_fix_tier_rank |
|
.pyqual/costs.json |
ai_cost |
|
| Linting | .pyqual/ruff.json |
ruff_errors, ruff_fatal, ruff_warnings |
.pyqual/pylint.json |
pylint_errors, pylint_fatal, pylint_error, pylint_warnings, pylint_score |
|
.pyqual/flake8.json |
flake8_violations, flake8_errors, flake8_warnings, flake8_conventions |
|
.pyqual/mypy.json |
mypy_errors |
|
| Documentation | .pyqual/interrogate.json |
docstring_coverage, docstring_total, docstring_missing |
Custom metrics: subclass MetricCollector and register with PluginRegistry โ see examples/custom_plugins/.
Gate operators
Metric key suffixes translate to comparison operators:
metrics:
cc_max: 15 # cc โค 15
coverage_min: 80 # coverage โฅ 80
critical_max: 0 # critical โค 0
error_count_max: 5 # error_count โค 5
vallm_pass_min: 90 # vallm_pass โฅ 90
mypy_errors_eq: 0 # mypy_errors = 0
| Suffix | Operator |
|---|---|
_max |
โค |
_min |
โฅ |
_lt |
< |
_gt |
> |
_eq |
= |
Integration with ecosystem
pyqual orchestrates โ it does not implement the analysis tools:
- code2llm โ code analysis โ pyqual reads
.toonoutput - vallm โ AI validation โ pyqual reads pass rates
- llx โ LLM routing, MCP workflows, issue parsing (requires Python โฅ 3.10)
- planfile โ ticket management โ pyqual syncs TODO.md and GitHub Issues
- costs โ AI spend tracking โ pyqual can gate on
ai_cost - algitex โ imports pyqual as a dependency for its
gocommand
Examples
See examples/ for ready-to-use configurations.
Basics
| Example | Description |
|---|---|
basic/ |
Python API โ Pipeline, GateSet, minimal one-liner |
python-package/ |
Standard Python package (src-layout) |
python-flat/ |
Flat project without src/ |
monorepo/ |
Multiple packages in one repository |
Quality & Linting
| Example | Tools |
|---|---|
linters/ |
ruff, pylint, flake8, mypy, interrogate |
security/ |
bandit, pip-audit, trufflehog, gitleaks, sbom |
custom_gates/ |
Dynamic thresholds, composite scoring |
custom_plugins/ |
Build your own MetricCollector plugins |
AI & LLM
| Example | Description |
|---|---|
llx/ |
Standalone llx integration โ model auto-selection |
llm_fix/ |
Docker-backed llx MCP fix workflow |
CI/CD
| Example | Platform |
|---|---|
github-actions/ |
GitHub Actions โ PR checks, artifacts, coverage |
gitlab-ci/ |
GitLab CI โ coverage reports, caching |
Advanced
| Example | Key Feature |
|---|---|
multi_gate_pipeline/ |
21-gate production pipeline (linters + security + AI + testing) |
ticket_workflow/ |
Auto TODO.md + GitHub Issues on gate failure |
project_analysis/ |
Gating on code2llm / toon analysis metrics |
Why not add this to algitex?
algitex has 29,448 lines, CCฬ=3.6, 64 critical issues, vallm pass 42.8%. Adding more features makes it worse. pyqual does one thing well: declarative quality gate loops. algitex imports pyqual. Both improve.
License
Licensed under Apache-2.0.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file pyqual-0.1.53.tar.gz.
File metadata
- Download URL: pyqual-0.1.53.tar.gz
- Upload date:
- Size: 638.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.13.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
827a61cd93734e2d466fd6b368debbdb14543bb2969c557c85230fd95d6b50db
|
|
| MD5 |
ff7841ee53a9da0473bf09c834593afc
|
|
| BLAKE2b-256 |
6a552c758ab535a5658f2b794b6b1648f9be1c7df3ebe4567c4c19959c05f845
|
File details
Details for the file pyqual-0.1.53-py3-none-any.whl.
File metadata
- Download URL: pyqual-0.1.53-py3-none-any.whl
- Upload date:
- Size: 72.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.13.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
72a4772cf9f57eca4663a552c1bbd223c263ba5b7b0309391b8bd6c87b334701
|
|
| MD5 |
7686166c69aa49cda5a4a62e46458ec2
|
|
| BLAKE2b-256 |
bd91f064f1bf92293fa053c28dc9ef6c037316305c9f2c109e858f46a2e6167d
|