Skip to main content

aspice-eval

ASPICE evaluation engine — knowledge base, gap analysis evaluator, and reports.

Library Usage

Evaluate an SDP

from aspice_eval import evaluate_sdp, ModelConfig

result = evaluate_sdp(
    "docs/sdp.md",
    ModelConfig(
        provider="bedrock",
        model_name="us.anthropic.claude-sonnet-4-20250514-v1:0",
        region="us-east-1",
    ),
    target_level=3,
    process_groups=["SWE", "SYS"],
)

print(f"Criteria assessed: {len(result.ratings)}")
print(f"Gaps found: {len([r for r in result.ratings if r.gaps])}")
print(f"Tokens used: {result.token_usage['total_tokens']}")

Validate a Knowledge Base

from aspice_eval import validate_kb

result = validate_kb("knowledge_base")

if not result.is_valid:
    for error in result.schema_errors:
        print(f"Schema error: {error}")
    for gap in result.completeness_gaps:
        print(f"Completeness gap: {gap}")
else:
    print("Knowledge base is valid")

Extension Points

Custom Evaluator

Subclass GapAnalysisEvaluator to plug in a custom LLM provider or rule-based engine:

from aspice_eval import GapAnalysisEvaluator, ModelConfig, register_evaluator

class LocalLlamaEvaluator(GapAnalysisEvaluator):
    """Evaluator using a local Llama model."""

    def _call_model(self, prompt: str) -> str:
        # Call your local model and return JSON response
        ...

# Register the custom provider
register_evaluator("local-llama", LocalLlamaEvaluator)

# Use it via the standard factory
from aspice_eval import create_evaluator

evaluator = create_evaluator(ModelConfig(provider="local-llama"))

Custom Knowledge Base Standards

Three levels of extensibility for non-ASPICE standards:

Level 1 — Custom YAML files (no code required):

Drop a new subdirectory under the KB root with YAML files conforming to the criteria schema:

knowledge_base/
├── aspice/          # Built-in ASPICE v4.0
└── iso26262/        # Your custom standard
    ├── _metadata.yaml
    └── functional_safety.yaml
from aspice_eval import evaluate_sdp, ModelConfig

result = evaluate_sdp(
    "docs/sdp.md",
    ModelConfig(provider="bedrock", model_name="...", region="us-east-1"),
    standard="iso26262",
)

Level 2 — In-memory construction via from_dict:

from aspice_eval import KnowledgeBase

kb = KnowledgeBase.from_dict({
    "processes": [
        {
            "process_id": "SWE.1",
            "process_name": "Software Requirements Analysis",
            "criteria": [...],
        }
    ]
})
criteria = kb.get_criteria(groups=["SWE"], max_level=3)

Level 3 — Custom KB loader (pluggable schema):

For standards with fundamentally different structures, subclass KnowledgeBase and register a loader:

from aspice_eval import KnowledgeBase, register_kb_loader, CriteriaEntry

class NISTCSFKnowledgeBase(KnowledgeBase):
    """Custom loader for NIST Cybersecurity Framework."""

    def load(self, standard: str) -> None:
        # Read NIST-shaped YAML/JSON, convert to CriteriaEntry list
        ...

    def get_criteria(self, groups, max_level) -> list[CriteriaEntry]:
        # Return entries filtered by NIST "Functions" instead of ASPICE groups
        ...

register_kb_loader("nist-csf", NISTCSFKnowledgeBase)

Custom Report Renderer

Subclass ReportRenderer to output evaluation results in formats beyond Markdown and HTML:

from aspice_eval import ReportRenderer, register_renderer
from aspice_eval import EvaluationResult, CapabilityLevelResult, EvaluationConfig

class JSONReportRenderer(ReportRenderer):
    """Render evaluation results as JSON."""

    def render(self, evaluation, levels, config, kb_metadata) -> str:
        import json
        return json.dumps({
            "ratings": [
                {"criteria_id": r.criteria_id, "rating": r.rating, "gaps": r.gaps}
                for r in evaluation.ratings
            ],
        }, indent=2)

# Register and use
register_renderer("json", JSONReportRenderer)

CLI Usage

aspice-eval evaluate

# Evaluate an SDP document
aspice-eval evaluate --sdp path/to/sdp.md --target-level 2 --groups SWE,MAN

# Write report to a file
aspice-eval evaluate --sdp path/to/sdp.md --output report.md

# Use a specific AI provider
aspice-eval evaluate --sdp path/to/sdp.md --provider bedrock \
  --model us.anthropic.claude-sonnet-4-20250514-v1:0 --region us-east-1

aspice-eval validate-kb

# Validate the default knowledge base
aspice-eval validate-kb

# Validate a custom knowledge base
aspice-eval validate-kb --kb-path /path/to/knowledge_base

Installation

pip install aspice-eval

# With AI provider support
pip install "aspice-eval[bedrock]"     # Amazon Bedrock (Claude)
pip install "aspice-eval[openai]"      # OpenAI GPT-4o
pip install "aspice-eval[anthropic]"   # Anthropic Claude (direct API)
pip install "aspice-eval[all]"         # All providers

Requires Python 3.10+.

License

MIT

Metadata

Release files for aspice-eval 0.2.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for aspice-eval 0.2.5
File Size Uploaded Provenance Actions
aspice_eval-0.2.5.tar.gz 88.8 kB Without provenance Details for aspice_eval-0.2.5.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for aspice-eval 0.2.5
File Interpreter ABI Platform Actions
aspice_eval-0.2.5-py3-none-any.whl Python 3 none any Details for aspice_eval-0.2.5-py3-none-any.whl

Total release size: 161.2 kB

Release files / aspice_eval-0.2.5.tar.gz

Download URL aspice_eval-0.2.5.tar.gz
Size 88.8 kB
Tags Source
SHA-256 checksum
How to use checksums
9d353f3224205161a3d82d3bacd4350ba061d9a1ec28ad341e300db91bca72a5
BLAKE2b-256 checksum
How to use checksums
62eb299ace715ced8ebae2221e607ad84ffb372feb539767885d90c0991bafc6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.11.15

Release files / aspice_eval-0.2.5-py3-none-any.whl

Download URL aspice_eval-0.2.5-py3-none-any.whl
Size 72.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
3b8a0a007240460f06f45627262b1a91162fdb446fb4e93c7a750c217b22bfd0
BLAKE2b-256 checksum
How to use checksums
c3d6b14946ab9eaec498d3aa0429f94086e18911885e6f19f87996752ff78e4e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.11.15

Security for aspice-eval 0.2.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page