aspice-eval
ASPICE evaluation engine — knowledge base, gap analysis evaluator, and reports.
Library Usage
Evaluate an SDP
from aspice_eval import evaluate_sdp, ModelConfig
result = evaluate_sdp(
"docs/sdp.md",
ModelConfig(
provider="bedrock",
model_name="us.anthropic.claude-sonnet-4-20250514-v1:0",
region="us-east-1",
),
target_level=3,
process_groups=["SWE", "SYS"],
)
print(f"Criteria assessed: {len(result.ratings)}")
print(f"Gaps found: {len([r for r in result.ratings if r.gaps])}")
print(f"Tokens used: {result.token_usage['total_tokens']}")
Validate a Knowledge Base
from aspice_eval import validate_kb
result = validate_kb("knowledge_base")
if not result.is_valid:
for error in result.schema_errors:
print(f"Schema error: {error}")
for gap in result.completeness_gaps:
print(f"Completeness gap: {gap}")
else:
print("Knowledge base is valid")
Extension Points
Custom Evaluator
Subclass GapAnalysisEvaluator to plug in a custom LLM provider or rule-based engine:
from aspice_eval import GapAnalysisEvaluator, ModelConfig, register_evaluator
class LocalLlamaEvaluator(GapAnalysisEvaluator):
"""Evaluator using a local Llama model."""
def _call_model(self, prompt: str) -> str:
# Call your local model and return JSON response
...
# Register the custom provider
register_evaluator("local-llama", LocalLlamaEvaluator)
# Use it via the standard factory
from aspice_eval import create_evaluator
evaluator = create_evaluator(ModelConfig(provider="local-llama"))
Custom Knowledge Base Standards
Three levels of extensibility for non-ASPICE standards:
Level 1 — Custom YAML files (no code required):
Drop a new subdirectory under the KB root with YAML files conforming to the criteria schema:
knowledge_base/
├── aspice/ # Built-in ASPICE v4.0
└── iso26262/ # Your custom standard
├── _metadata.yaml
└── functional_safety.yaml
from aspice_eval import evaluate_sdp, ModelConfig
result = evaluate_sdp(
"docs/sdp.md",
ModelConfig(provider="bedrock", model_name="...", region="us-east-1"),
standard="iso26262",
)
Level 2 — In-memory construction via from_dict:
from aspice_eval import KnowledgeBase
kb = KnowledgeBase.from_dict({
"processes": [
{
"process_id": "SWE.1",
"process_name": "Software Requirements Analysis",
"criteria": [...],
}
]
})
criteria = kb.get_criteria(groups=["SWE"], max_level=3)
Level 3 — Custom KB loader (pluggable schema):
For standards with fundamentally different structures, subclass KnowledgeBase and register a loader:
from aspice_eval import KnowledgeBase, register_kb_loader, CriteriaEntry
class NISTCSFKnowledgeBase(KnowledgeBase):
"""Custom loader for NIST Cybersecurity Framework."""
def load(self, standard: str) -> None:
# Read NIST-shaped YAML/JSON, convert to CriteriaEntry list
...
def get_criteria(self, groups, max_level) -> list[CriteriaEntry]:
# Return entries filtered by NIST "Functions" instead of ASPICE groups
...
register_kb_loader("nist-csf", NISTCSFKnowledgeBase)
Custom Report Renderer
Subclass ReportRenderer to output evaluation results in formats beyond Markdown and HTML:
from aspice_eval import ReportRenderer, register_renderer
from aspice_eval import EvaluationResult, CapabilityLevelResult, EvaluationConfig
class JSONReportRenderer(ReportRenderer):
"""Render evaluation results as JSON."""
def render(self, evaluation, levels, config, kb_metadata) -> str:
import json
return json.dumps({
"ratings": [
{"criteria_id": r.criteria_id, "rating": r.rating, "gaps": r.gaps}
for r in evaluation.ratings
],
}, indent=2)
# Register and use
register_renderer("json", JSONReportRenderer)
CLI Usage
aspice-eval evaluate
# Evaluate an SDP document
aspice-eval evaluate --sdp path/to/sdp.md --target-level 2 --groups SWE,MAN
# Write report to a file
aspice-eval evaluate --sdp path/to/sdp.md --output report.md
# Use a specific AI provider
aspice-eval evaluate --sdp path/to/sdp.md --provider bedrock \
--model us.anthropic.claude-sonnet-4-20250514-v1:0 --region us-east-1
aspice-eval validate-kb
# Validate the default knowledge base
aspice-eval validate-kb
# Validate a custom knowledge base
aspice-eval validate-kb --kb-path /path/to/knowledge_base
Installation
pip install aspice-eval
# With AI provider support
pip install "aspice-eval[bedrock]" # Amazon Bedrock (Claude)
pip install "aspice-eval[openai]" # OpenAI GPT-4o
pip install "aspice-eval[anthropic]" # Anthropic Claude (direct API)
pip install "aspice-eval[all]" # All providers
Requires Python 3.10+.
License
MIT
Metadata
Release files for aspice-eval 0.2.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| aspice_eval-0.2.5.tar.gz | 88.8 kB | Without provenance | Details for aspice_eval-0.2.5.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| aspice_eval-0.2.5-py3-none-any.whl | Python 3 | none | any | Details for aspice_eval-0.2.5-py3-none-any.whl |
Total release size: 161.2 kB
Release files / aspice_eval-0.2.5.tar.gz
| Download URL | aspice_eval-0.2.5.tar.gz |
|---|---|
| Size | 88.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9d353f3224205161a3d82d3bacd4350ba061d9a1ec28ad341e300db91bca72a5
|
|
BLAKE2b-256 checksum How to use checksums |
62eb299ace715ced8ebae2221e607ad84ffb372feb539767885d90c0991bafc6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.15
|
Release files / aspice_eval-0.2.5-py3-none-any.whl
| Download URL | aspice_eval-0.2.5-py3-none-any.whl |
|---|---|
| Size | 72.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
3b8a0a007240460f06f45627262b1a91162fdb446fb4e93c7a750c217b22bfd0
|
|
BLAKE2b-256 checksum How to use checksums |
c3d6b14946ab9eaec498d3aa0429f94086e18911885e6f19f87996752ff78e4e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.15
|
Security for aspice-eval 0.2.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.