sfi-toolkit

sfi-toolkit is a Python library for Semantic Fault Injection (SFI) — a testing methodology for LLM-based agent resilience.

Features

• Rule-based semantic fault generator — deterministically perturbs correct tool responses into wrong-period, wrong-entity, or numeric-noise faults. All faults are schema-valid and statistically plausible.
• Three hardening strategies — H1 (cross-source validation), H2 (business-rule plausibility), H3 (trajectory consistency) — independently and composably reduce fault propagation.
• Propagation experiment runner — simulates the agent as a Bernoulli chain and measures fault propagation rates across fault classes, injection steps, and hardening configurations.
• Provider-agnostic LLM fault generator — accepts any Callable[[str], str] as the LLM backend; no SDK dependency.

Installation

pip install sfi-toolkit

Built for Python 3.12 or above.

Quick Start

1. Generate a wrong-period fault

from sfi_toolkit import FaultClass, RuleBasedFaultGenerator

schema = {
    "type": "object",
    "required": ["company_code", "fiscal_period", "accounts"],
    "properties": {
        "company_code": {"type": "string"},
        "fiscal_period": {"type": "string"},
        "accounts": {"type": "array"},
    },
}

correct_response = {
    "company_code": "1000",
    "fiscal_period": "2025-Q3",
    "accounts": [{"account_id": "110000", "balance": 1842350.00}],
}

gen = RuleBasedFaultGenerator(
    schema=schema,
    entity_catalog=["1000", "2000", "3000", "4000", "5000"],
)
fault = gen.generate(correct_response, FaultClass.WRONG_PERIOD)
print(fault.perturbed_response["fiscal_period"])  # "2025-Q2"
print(fault.schema_valid)                          # True

2. Check a response with H3 trajectory consistency

from sfi_toolkit import H3Consistency

correct_response = {
    "company_code": "1000",
    "fiscal_period": "2025-Q3",
    "accounts": [{"account_id": "110000", "balance": 1842350.00}],
}

h3 = H3Consistency()
ctx = {"company_code": "1000", "fiscal_period": "2025-Q3"}

h3.observe(correct_response, ctx)

wrong = dict(correct_response)
wrong["fiscal_period"] = "2025-Q2"
result = h3.check(wrong, ctx)
print(result.detected)  # True
print(result.reason)    # "fiscal_period mismatch: ..."

3. Run a propagation experiment

from sfi_toolkit import FaultClass, RuleBasedFaultGenerator, PropagationExperiment

schema = {
    "type": "object",
    "required": ["company_code", "fiscal_period", "accounts"],
    "properties": {
        "company_code": {"type": "string"},
        "fiscal_period": {"type": "string"},
        "accounts": {"type": "array"},
    },
}

correct_response = {
    "company_code": "1000",
    "fiscal_period": "2025-Q3",
    "accounts": [{"account_id": "110000", "balance": 1842350.00}],
}

gen = RuleBasedFaultGenerator(
    schema=schema,
    entity_catalog=["1000", "2000", "3000", "4000", "5000"],
)
faults = gen.generate_batch(correct_response, FaultClass.WRONG_PERIOD, n=50)

exp = PropagationExperiment()
result_h0 = exp.run(faults, injection_step=1, hardening=set())
result_all = exp.run(faults, injection_step=1, hardening={"H1", "H2", "H3"})

print(f"H0  propagation rate: {result_h0.propagation_rate:.2f}")
print(f"All propagation rate: {result_all.propagation_rate:.2f}")

Fault Taxonomy

SFI targets the environment-side innocent cell — the gap not addressed by any prior methodology.

API Reference

See API.md.

License

MIT License. See LICENSE.