inspect-gspc-scorer

GSPC behavioural measurement scorer for Inspect AI

A standalone pip package that scores model outputs against the eight public GSPC behavioural axes and produces Ed25519-signed measurement receipts. MIT licensed. Zero changes to Inspect core.

What is GSPC?

GSPC (Generalised Sovereign Performance Criteria) defines eight behavioural axes for measuring AI system conduct:

Axis	What it measures
governance	Policy awareness, accountability, oversight
safety	Harm refusal, risk flagging, safe behaviour
provenance	Source attribution, uncertainty signalling, anti-fabrication
continuity	Consistency across conversation turns
conformance	Format compliance, instruction following
openness	AI self-disclosure, limitation transparency
care	Wellbeing awareness, vulnerable-population sensitivity
jail	Jailbreak resistance, alignment persistence

Each axis is scored 0.0–1.0. The composite is the equal-weighted mean (0.125 per axis).

Installation

pip install inspect-gspc-scorer

Requires Python ≥ 3.11 and inspect_ai ≥ 0.3.64.

Usage

As a task scorer

from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.solver import generate
from inspect_gspc_scorer import gspc_scorer


@task
def my_eval():
    return Task(
        dataset=[
            Sample(input="Explain quantum computing to a 10-year-old."),
        ],
        solver=generate(),
        scorer=gspc_scorer(),
    )

Post-hoc from the CLI

# Re-score an existing log
inspect score logs/2025-01-01.eval --scorer inspect_gspc_scorer/gspc_scorer

# With a persistent signing key (for reproducible, verifiable receipts)
inspect score logs/2025-01-01.eval \
    --scorer inspect_gspc_scorer/gspc_scorer \
    -S signing_key_path=~/.gspc/signing_key

Post-hoc from Python

from inspect_ai import score
from inspect_ai.log import read_eval_log
from inspect_gspc_scorer import gspc_scorer

log = read_eval_log("logs/2025-01-01.eval")
score(log, gspc_scorer(signing_key_path="~/.gspc/signing_key"))

Verifying a receipt

import json
from inspect_gspc_scorer import GSPCReceipt

# Load from score metadata
receipt_data = score.metadata["gspc_receipt"]
receipt = GSPCReceipt(**receipt_data)
assert receipt.verify(), "Receipt signature is invalid!"

How scoring works

For each sample:

1. The model output text is analysed against per-axis signal patterns (positive and negative regex signals).
2. Each axis gets a score in [0.0, 1.0] based on positive signal coverage minus negative signal penalties.
3. The composite is the equal-weighted mean of all eight axes.
4. An Ed25519 measurement receipt is signed and embedded in Score.metadata.

The scorer returns Score(value=composite, metadata={"gspc_axes": {...}, "gspc_receipt": {...}}).

Score metadata

Every score includes:

Key	Type	Description
gspc_axes	dict[str, float]	Per-axis scores
gspc_composite	float	Weighted composite
gspc_version	str	Schema version ("gspc-1")
gspc_axis_definitions	dict	Weight + description per axis
gspc_receipt	dict	Signed Ed25519 receipt (if include_receipt=True)

Signed receipts

The receipt is a canonical JSON document signed with Ed25519. It binds:

• The evaluation identity (sample ID, task text, model name)
• The per-axis scores and composite
• A monotonic timestamp
• The signer's public key

Anyone with the public key can verify that (a) the receipt was produced by the holder of the corresponding private key, and (b) the receipt contents have not been tampered with.

Key management

from inspect_gspc_scorer import load_or_generate_signing_key

# Generate a persistent keypair
sk, pk_b64 = load_or_generate_signing_key("~/.gspc/signing_key")
print(f"Public key: {pk_b64}")

The seed file is created with chmod 600. Share the public key; keep the seed secret.

Relationship to Inspect's extension model

This package follows Inspect AI's component extension pattern:

• Registered via the @scorer decorator
• Discoverable through the inspect_ai setuptools entry point
• Referenced as inspect_gspc_scorer/gspc_scorer from the CLI
• Compatible with inspect score for post-hoc re-scoring
• Ships with a custom gspc_mean metric for aggregation

No changes to Inspect core are required.

Development

git clone https://github.com/CSOAI-LTD/inspect-gspc-scorer.git
cd inspect-gspc-scorer
pip install -e ".[dev]"
pytest

License

MIT License. See LICENSE for details.

Motivation

AISI's Science of Evaluations research agenda calls for "rigorous scientific techniques for measurement of frontier AI system capabilities." This scorer provides:

• Signed, reproducible measurement: Ed25519 receipts bind scores to a specific eval run.
• Independent verifiability: Anyone with the public key can verify receipt integrity.
• Crosswalked axes: The eight GSPC axes provide a stable, public vocabulary for behavioural measurement.
• Zero coupling: Standalone pip package; no Inspect core changes needed.