syncreus-eval 0.1.0

Open-source AI evaluation toolkit — hallucination detection, safety, industry-specific evals

syncreus-eval

Open-source AI evaluation toolkit for hallucination detection, safety scanning, bias analysis, and industry-specific evals. Runs locally without the Syncreus platform.

Installation

# Core (LLM-as-judge evaluators via Gemini)
pip install syncreus-eval

# With optional extras
pip install syncreus-eval[accuracy]          # fastembed for semantic similarity
pip install syncreus-eval[safety]            # Presidio PII scanning
pip install syncreus-eval[prompt-injection]  # LLM Guard injection detection
pip install syncreus-eval[upload]            # Upload results to Syncreus platform
pip install syncreus-eval[all]               # Everything

Quick Start

from syncreus_eval import evaluate, EvalType

# Hallucination detection
result = evaluate(
    EvalType.HALLUCINATION,
    ai_input="The Eiffel Tower is in Paris, France. It was built in 1889.",
    ai_output="The Eiffel Tower is in Paris, France. It was built in 1889 and is 300 meters tall.",
    gemini_key="your-gemini-api-key",
)
print(result.passed)    # True/False/None
print(result.details)   # Claim-level verdicts

# Performance tracking (no LLM needed)
result = evaluate(
    EvalType.PERFORMANCE,
    trace={"latency_ms": 150, "token_count_input": 100, "token_count_output": 50},
)
print(result.details)   # {"latency_ms": 150, "total_tokens": 150, ...}

# Run multiple evals at once
results = evaluate(
    [EvalType.HALLUCINATION, EvalType.IDEOLOGY],
    ai_input="Context here",
    ai_output="Response here",
    gemini_key="your-key",
)
for r in results:
    print(f"{r.eval_type.value}: passed={r.passed}")

Evaluation Types

General Purpose

Type	Description	Requires
HALLUCINATION	Detects unsupported factual claims	Gemini API key
ACCURACY	Golden dataset comparison via semantic similarity	[accuracy] extra
CONSISTENCY	Pairwise similarity across repeated prompts	[accuracy] extra
PERFORMANCE	Extracts latency, tokens, cost metrics	Nothing
AGENT_TASK	Verifies agent completion claim honesty	Gemini API key
REGRESSION	Baseline comparison (platform only)	Syncreus platform

Safety & Compliance

Type	Description	Requires
SAFETY	PII/sensitive data detection + content safety	[safety] extra
BIAS	Demographic parity / EEOC four-fifths rule	Nothing
IDEOLOGY	Political neutrality (OMB M-26-04)	Gemini API key
PROMPT_INJECTION	Injection attempt detection	[prompt-injection] extra

Industry-Specific

Type	Description	Requires
HEALTHCARE	Medical accuracy, drug safety, PHI detection	Gemini API key
LEGAL	Citation validity, holding fidelity	Gemini API key
FINANCE	Regulatory accuracy, numerical precision	Gemini API key
CODE_ACCURACY	API existence, function signatures	Gemini API key

API Reference

evaluate()

from syncreus_eval import evaluate, EvalType

result = evaluate(
    eval_type=EvalType.HALLUCINATION,  # or a list of types
    ai_input="...",
    ai_output="...",
    gemini_key="...",                  # or set GEMINI_API_KEY env var
    # Accuracy-specific:
    test_cases=[{"input_text": "...", "expected_output": "..."}],
    threshold=0.85,
    # Performance-specific:
    trace={"latency_ms": 100, ...},
    # Agent task-specific:
    verification_result="exit code 0",
    # Bias-specific:
    traces=[{"metadata": {"demographic_group": "A"}, "passed": True}],
    # Consistency-specific:
    outputs=["response1", "response2", "response3"],
    # Safety-specific:
    entity_whitelist=["aspirin"],
    enable_gemini_content_safety=True,
)

Returns an EvalResult (or list of them):

class EvalResult:
    eval_type: EvalType
    passed: bool | None      # True/False/None (None = error or skipped)
    score: float | None       # Numeric score where applicable
    details: dict[str, Any]   # Evaluator-specific details
    error: bool               # Whether an error occurred
    error_message: str | None # Error description

upload_results() (optional)

from syncreus_eval import upload_results

upload_results(
    results=result,           # EvalResult or list
    api_key="syn_...",        # Syncreus API key
    endpoint="https://api.syncreus.com",
    trace_id="trace-123",     # optional
)

Requires: pip install syncreus-eval[upload]

Environment Variables

Variable	Description
GEMINI_API_KEY	Google Gemini API key for LLM-as-judge evaluators

License

MIT