defect-check is a standalone, framework-free inspection engine for AI artifacts. It accepts Skills, Tools, and Prompts as input, runs multi-dimensional quality checks, and returns structured defect reports with scores and severity ratings.

• 🔍 Five inspection modules — QDS, QDT, QDP, Cross (PS/PT/ST), and QD-PM (Permission)
• 🤖 LLM-powered analysis — supports OpenAI, Anthropic, and DashScope providers
• 🎚️ Three inspection levels — L1 (quick), L2 (standard), L3 (deep)
• 🛡️ Permission risk assessment — OR-L1/L2/L3 operation risk levels for safety-critical checks
• ⚖️ Built-in rate control — configurable retry (default 3) and LLM concurrency limit (default 5)
• 📦 Zero infrastructure — no database, no API server, no task queue
• 🏗️ Framework-free — bring your own runtime, the engine stays pure

Table of Contents

• defect-check
  • Table of Contents
  • Installation
  • Quick Start
    • Recommended: the DefectChecker class
  • Inspection Levels
  • LLM Configuration
    • Retry and Concurrency Control
    • Custom YAML Configuration
  • Inspection Modules
  • Response Format
    • Defect Fields
  • API Reference
    • defect_check.check(...)
    • defect_check.check_single(...)
    • defect_check.check_cross(...)
    • DefectChecker class
    • Exported Types
  • Development
  • Contributing
  • License

Installation

pip install defect-check

Requires Python ≥ 3.11

Quick Start

import asyncio
import defect_check

async def main():
    result = await defect_check.check(
        tools=[
            {
                "name": "lookup_order",
                "description": "Query orders by order ID",
                "parameters": {"type": "object"},
            }
        ],
        prompts=[
            {"name": "system", "content": "You are an order assistant."}
        ],
        skills=[
            {"id": "orders", "name": "orders", "content": "# Orders workflow"}
        ],
        llm_provider="openai",
        llm_base_url="https://api.example.com/v1",
        llm_api_key="your-api-key",
        llm_model_id="your-model-id",
    )
    print(result)

asyncio.run(main())

Recommended: the DefectChecker class

For repeated use, create a DefectChecker once with your LLM settings and reuse it — no need to pass base parameters on every call. All check methods are async.

import asyncio
from defect_check import DefectChecker

async def main():
    # Configure everything up front (max_retries defaults to 3, max_concurrency to 5)
    checker = DefectChecker(
        llm_provider="openai",
        llm_base_url="https://api.example.com/v1",
        llm_api_key="your-api-key",
        llm_model_id="your-model-id",
        max_retries=3,          # retries for recoverable LLM errors
        max_concurrency=5,      # max in-flight LLM requests across all calls
    )

    result = await checker.defect_check(
        tools=[{"name": "lookup_order", "description": "Query orders by order ID"}],
        prompts=[{"name": "system", "content": "You are an order assistant."}],
        skills=[{"id": "orders", "name": "orders", "content": "# Orders workflow"}],
    )

    single = await checker.check_single(tool={"name": "lookup_order"})
    cross = await checker.cross_defect_check(prompt="You are...", skills=skills, tools=tools)

    await checker.close()

asyncio.run(main())

You can also create the checker first and configure it later via setter methods:

checker = DefectChecker()
checker.set_llm_config(llm_provider="openai", llm_api_key="sk-...", llm_model_id="gpt-4o")
checker.set_max_retries(5)      # override the default of 3
checker.set_max_concurrency(10) # override the default of 5

result = await checker.defect_check(tools=tools)

Inspection Levels

Use check_level to control inspection depth:

result = await defect_check.check(
    tools=tools,
    prompts=prompts,
    skills=skills,
    check_level="L3",
    llm_provider="openai",
    llm_api_key="your-api-key",
    llm_model_id="your-model-id",
)

When omitted, QDS determines the level using the bundled checklist, while QDT, QDP, and Cross determine it via the LLM. You can also pass check_level through options:

result = await defect_check.check(
    tools=tools,
    prompts=prompts,
    skills=skills,
    options=defect_check.DefectCheckOptions(check_level="L2"),
    llm_provider="openai",
    llm_api_key="your-api-key",
    llm_model_id="your-model-id",
)

The legacy qdp_check_level option remains supported for backwards compatibility. Conflicting values (e.g. check_level="L3" + options={"qdp_check_level": "L1"}) will raise an error.

Permission Check Options

Two additional options in DefectCheckOptions control the QD-PM permission inspection:

Invalid or_level values (anything other than OR-L1/OR-L2/OR-L3/None) raise a ValueError.

LLM Configuration

LLM settings are passed explicitly by the caller — the package does not read .env files or environment variables for LLM configuration.

You can also pass a pre-configured client object via the provider parameter, bypassing the four llm_* parameters:

from defect_check.llm import DefectCheckTextClient

# Build your own client, then pass it in
client = DefectCheckTextClient(my_custom_provider)

result = await defect_check.check(
    tools=tools,
    prompts=prompts,
    skills=skills,
    provider=client,
)

The same llm_* / provider parameters are accepted by DefectChecker(...) and checker.set_llm_config(...).

Retry and Concurrency Control

The LLM client applies two protections that you can tune (defaults shown):

• max_retries — recoverable errors (HTTP 429, provider 5xx, connection failures) are retried with exponential backoff. Default: 3.
• max_concurrency — an instance-wide cap on simultaneous in-flight LLM requests, shared across every check call made on the same checker. The semaphore covers each request and its retries. Default: 5.

checker = DefectChecker(provider=my_client, max_retries=5, max_concurrency=10)

Best practice: set max_concurrency before running your first check. Changing it afterwards rebuilds the underlying client's shared semaphore, which briefly releases coordination for in-flight callers.

If the LLM configuration is missing when a check runs, the response carries an LLM_CONFIGURATION_MISSING error instead of raising an exception.

Custom YAML Configuration

You can override the built-in inspection rules and prompt templates with custom YAML files:

checker = DefectChecker(
    llm_provider="openai",
    llm_api_key="your-api-key",
    llm_model_id="gpt-4o",
    yaml_overrides={
        "qds_scoring": "/path/to/custom_qds_rules.yaml",
        "prompt_qds": "/path/to/custom_qds_prompt.yaml",
    },
)

# Or switch at runtime
checker.set_yaml_overrides({"qds_scoring": "/another/config.yaml"})

# Reset to built-in defaults
checker.set_yaml_overrides(None)

Supported override keys:

For detailed YAML schema, domain-specific extensions, and examples, see Custom YAML Configuration.

Inspection Modules

The package provides five inspection modules, each targeting a different artifact dimension:

Every supplied Skill, Tool, and Prompt is inspected. The response always uses a consistent envelope — results is always a list: one input produces one result item, multiple inputs produce multiple result items.

Inspection rules and prompt templates are packaged in the wheel. QDT, QDP, and Cross load YAML resources; QDS loads the bundled checklist.py.

Permission Inspection (QD-PM)

QD-PM is a safety-focused module that inspects permission and operation risks in AI artifacts. It runs after single-artifact and cross-artifact inspections, subject to Gate-0 pre-validation.

Operation Risk Levels (OR-L):

Gate-0 Pre-validation:

Before QD-PM runs, Gate-0 validates four structural preconditions:

• G0-1: No unfixed P0 defects in single-artifact inspection (QDS/QDT/QDP)
• G0-2: No unfixed P0 defects in cross-artifact inspection
• G0-3: A responsibility-boundary statement exists and is locatable
• G0-4: Every Tool Schema has sufficient spec for operational semantics

If any precondition fails, QD-PM returns an "evaluation cannot be performed" signal instead of a defect list.

Configuration:

result = await defect_check.check(
    tools=tools,
    prompts=prompts,
    skills=skills,
    options=defect_check.DefectCheckOptions(
        enable_permission_check=True,  # enable QD-PM (default: True)
        or_level="OR-L2",              # set operation risk level (OR-L1/OR-L2/OR-L3)
    ),
    llm_provider="openai",
    llm_api_key="your-api-key",
    llm_model_id="gpt-4o",
)

Unified Inspection Framework

Since v2.x, QDS, QDT, and QDP share a single unified inspection framework in defect_check/core/. Each domain is a thin declarative registration (DomainRegistration) wired into a 7-stage pipeline:

LevelDeterminer → EntryGate → ViewGenerator → Checklist → CheckExecutor → Enricher → Scorer → Adapter

All three domains feed the same InspectionPipeline (see defect_check/orchestrator.py), which is what the public check / check_single entry points call. This guarantees identical contract behavior across domains while keeping QDS-specific features (QDS-0 gate, intermediate view, grouped parallel execution) as first-class framework concepts.

The orchestrator also runs QD-PM (Permission Inspection) after the single-artifact and cross-artifact stages, gated by Gate-0 pre-validation. QD-PM inspects permission proportionality using operation risk levels (OR-L) and agent capability levels (AC-L).

To add a new domain, implement a DomainRegistration (level determiner, gate, view generator, executor, enricher, scorer) and register it in DomainRegistry — see docs/architecture.md for details.

Response Format

{
  "schema_version": "1.0",
  "status": "completed",
  "results": [
    {
      "module": "QDT",
      "check_type": "artifact",
      "status": "completed",
      "check_level": "L2",
      "artifacts": [{"type": "tool", "id": "lookup_order", "name": "lookup_order"}],
      "score": {"total_score": 100.0, "max_score": 100.0, "grade": null, "gate_result": "PASS"},
      "defect_summary": {"total_defects": 0, "p0_count": 0, "p1_count": 0, "p2_count": 0},
      "defects": [],
      "error": null,
      "details": {},
      "metadata": {}
    }
  ],
  "summary": {
    "total_results": 1,
    "completed_results": 1,
    "failed_results": 0,
    "skipped_results": 0,
    "total_defects": 0,
    "p0_count": 0,
    "p1_count": 0,
    "p2_count": 0,
    "gate_result": "PASS"
  },
  "errors": [],
  "metadata": {"execution_time_seconds": 0.0}
}

Defect Fields

Each defect in the defects list contains these canonical fields:

API Reference

defect_check.check(...)

Inspect caller-provided Skills, Tools, and Prompts.

async def check(
    tools: list[dict] | None,
    prompts: list[dict] | None,
    skills: list[dict] | None,
    *,
    check_level: str | None = None,
    options: DefectCheckOptions | dict | None = None,
    provider: Any | None = None,
    llm_provider: str | None = None,
    llm_base_url: str | None = None,
    llm_api_key: str | None = None,
    llm_model_id: str | None = None,
) -> dict[str, Any]

defect_check.check_single(...)

Inspect a single artifact. See the API documentation for details.

defect_check.check_cross(...)

Run cross-artifact inspection (PS/PT/ST). See the API documentation for details.

DefectChecker class

A stateful facade that holds LLM configuration, retry, and concurrency settings — configure once, then run checks as instance methods.

from defect_check import DefectChecker

checker = DefectChecker(
    provider: Any | None = None,          # pre-configured client/provider (optional)
    llm_provider: str | None = None,
    llm_base_url: str | None = None,
    llm_api_key: str | None = None,
    llm_model_id: str | None = None,
    max_retries: int = 3,
    max_concurrency: int = 5,
    options: DefectCheckOptions | dict | None = None,  # instance-level defaults
    yaml_overrides: dict[str, str] | None = None,      # custom YAML config overrides
)

# Configuration setters (can be called after construction)
checker.set_llm_config(llm_provider=..., llm_api_key=..., llm_model_id=...)
checker.set_max_retries(5)
checker.set_max_concurrency(10)
checker.set_options({"check_level": "L2"})
checker.set_yaml_overrides({"qds_scoring": "/path/to/custom.yaml"})  # custom rules

# Inspection methods (all async; parameter signatures match the module-level functions)
await checker.defect_check(tools=..., prompts=..., skills=...)   # full inspection
await checker.check_single(tool=..., prompt=..., skill=...)      # single artifact
await checker.cross_defect_check(prompt=..., skills=..., tools=...)  # PS/PT/ST only
await checker.close()                                            # release the LLM connection

Precedence for check settings: explicit method arguments > instance configuration (set_options / constructor options) > built-in defaults. Call-level check_level overrides the instance level without conflict errors.

Exported Types

from defect_check import (
    DefectCheckOptions,
    DefectCheckResponse,
    DefectItem,
    DefectSummary,
    InspectionResult,
    InspectionError,
    ScoreResult,
    ResponseSummary,
    ArtifactReference,
    SkillArtifact,
    PromptArtifact,
    RESOURCE_ALIASES,  # supported YAML override keys
)

Development

# Clone the repository
git clone https://github.com/sanityops-org/artifact-defect-check.git
cd artifact-defect-check

# Create a virtual environment
python -m venv .venv && source .venv/bin/activate

# Install in development mode
pip install -e ".[dev]"

# Run tests
pytest

Note: The package uses pytest with asyncio_mode = "strict" — async tests must be decorated with @pytest.mark.asyncio and awaited. When adding a new test module, keep its filename unique across the whole tests/ tree (pytest errors on two test files sharing the same basename in different directories).

Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Please make sure to update tests as appropriate and adhere to the existing code style.

License

This project is licensed under the Apache License 2.0 — see the LICENSE file for details.