ARGUS-AI

Heuristic quality scoring and monitoring for LLM outputs, in 3 lines of code

ARGUS-AI is the G-ARVIS scoring engine for monitoring LLM application quality. It scores each LLM response across six dimensions: Groundedness, Accuracy, Reliability, Variance, Inference Cost, and Safety.

Status: Beta (0.1.x). All scorers are lightweight rule-based heuristics (keyword, regex, and token-overlap checks), not trained models. Treat the scores as trend signals to monitor over time, not as verdicts on any single response. This repository does not include benchmarks of the scorers against labelled datasets.

import argus_ai

argus = argus_ai.init()
result = argus.evaluate(prompt=prompt, response=response, context=context)

That's it. Every LLM call you pass through evaluate gets a G-ARVIS score.

Why ARGUS

LLM outputs can degrade without raising errors: models update, prompts drift, and costs creep. Traditional monitoring catches latency and errors, but not changes in output quality. G-ARVIS gives you per-dimension scores you can export to your metrics stack and alert on, so a drop shows up in a dashboard instead of in a user complaint.

Plus three agentic evaluation metrics for autonomous workflows (scores in [0, 1]; see src/argus_ai/scoring/agentic.py for the exact formulas):

Install

pip install argus-llm

The PyPI package is named argus-llm; the Python import name is argus_ai. (The similarly named argus-ai package on PyPI is an unrelated project.)

With provider integrations:

pip install "argus-llm[anthropic]"      # Anthropic Claude wrapper
pip install "argus-llm[openai]"         # OpenAI wrapper
pip install "argus-llm[prometheus]"     # Prometheus export
pip install "argus-llm[opentelemetry]"  # OTEL export
pip install "argus-llm[all]"            # Everything

Quick Start

Basic Evaluation

import argus_ai

argus = argus_ai.init(profile="enterprise")

result = argus.evaluate(
    prompt="What causes climate change?",
    response="Greenhouse gas emissions from fossil fuels are the primary driver.",
    context="Climate change is driven by human activities releasing greenhouse gases.",
    model_name="claude-sonnet-4",
    latency_ms=1200.0,
    input_tokens=45,
    output_tokens=30,
    cost_usd=0.002,
)

print(f"Composite: {result.garvis_composite:.3f}")  # 0.755
print(f"Passing:   {result.passing}")                # True
print(f"Safety:    {result.safety:.3f}")             # 1.000

The values in the comments are the actual output of this snippet on argus-llm 0.1.1. Scores depend on your inputs and the heuristics, and may change between releases. (Here the groundedness score is low, 0.167, because the response shares few words with the context; the heuristic measures word overlap, not meaning.)

Agentic Workflow Evaluation

from argus_ai.types import AgenticEvalRequest

argus = argus_ai.init(profile="agentic")

workflow = AgenticEvalRequest(
    prompt="Research competitors and generate report",
    response="Report generated with 5 competitor analyses.",
    steps_planned=8,
    steps_completed=7,
    steps_failed=2,
    steps_recovered=1,
    retries=3,
    total_cost_usd=0.45,
)

result, agentic_metrics = argus.evaluate_agentic(workflow)

for m in agentic_metrics:
    print(f"{m.name}: {m.score:.3f}")
# AgentStabilityFactor: 0.632
# ErrorRecoveryRate: 0.540
# CostPerCompletedStep: 0.357

(Actual output on argus-llm 0.1.1.)

Drop-In Provider Wrappers

Anthropic Claude:

from argus_ai.integrations.anthropic import InstrumentedAnthropic

argus = argus_ai.init()
client = InstrumentedAnthropic(argus=argus)

response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Explain transformers"}],
)

# G-ARVIS score automatically attached
print(response._argus_score.garvis_composite)

OpenAI:

from argus_ai.integrations.openai import InstrumentedOpenAI

client = InstrumentedOpenAI(argus=argus_ai.init())

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Explain transformers"}],
)

print(response._argus_score.garvis_composite)

Threshold Monitoring with Alerting

from argus_ai.monitoring.thresholds import ThresholdConfig
from argus_ai.monitoring.alerts import AlertRule, AlertSeverity

config = ThresholdConfig(
    composite_min=0.80,
    safety_min=0.90,
    window_size=100,
    breach_ratio=0.15,
)

rules = [
    AlertRule(
        dimension="safety",
        threshold=0.85,
        severity=AlertSeverity.CRITICAL,
        message="Safety below critical threshold",
    ),
]

argus = argus_ai.init(
    profile="healthcare",
    thresholds=config,
    alert_rules=rules,
    exporters=["console", "prometheus"],
    on_alert=lambda msg, result: pagerduty.trigger(msg),
)

Decorator Instrumentation

from argus_ai.sdk.decorators import argus_evaluate

argus = argus_ai.init()

@argus_evaluate(argus, model_name="claude-sonnet-4")
def generate(prompt: str) -> str:
    return anthropic_client.messages.create(
        model="claude-sonnet-4-20250514",
        messages=[{"role": "user", "content": prompt}]
    ).content[0].text

Weight Profiles

G-ARVIS weights are configurable per deployment scenario:

These profiles only change how the six dimension scores are weighted into the composite. This library scores LLM outputs and does not provide or certify regulatory compliance.

Custom weights:

from argus_ai.scoring.garvis import GarvisWeights

argus = argus_ai.init(weights=GarvisWeights(safety=0.40, accuracy=0.30))

Metrics Export

Prometheus

argus = argus_ai.init(exporters=["prometheus"])

Exposes: argus_garvis_composite, argus_garvis_{dimension}, argus_evaluation_duration_ms, argus_alerts_total

OpenTelemetry

argus = argus_ai.init(exporters=["opentelemetry"])

Emits metrics through the OpenTelemetry API, so they flow to whatever exporter you configure in the OpenTelemetry SDK (for example OTLP). It has not been tested against specific vendor backends.

Architecture

See ARCHITECTURE.md for the full system design and open core split.

argus-ai (this package, Apache-2.0)   ARGUS Platform (planned; NOT part of
                                      this package and not available)
├── G-ARVIS heuristic scorers         ├── Autonomous Correction Loop
├── 3-Line SDK                        ├── Prompt Optimizer
├── Threshold Monitor                 ├── LLM-as-Judge Evaluation
├── ASF/ERR/CPCS Metrics              ├── Multi-Run Variance Analysis
├── Prometheus/OTEL Export            └── Dashboard UI
└── Anthropic/OpenAI Wrappers

Nothing in the right-hand column is included in, or can be obtained through, this package.

Performance

G-ARVIS scoring is pure Python (regex and string operations) with no network calls. Measured on one machine (Intel Xeon E3-1505M v6 @ 3.0 GHz, 64 GB RAM, Windows 10, Python 3.11, argus-llm 0.1.1, default console logging on, evaluating the Quick Start example repeatedly):

Your numbers will vary with hardware, input length, and logging configuration. Reproduce the timings with python benchmarks/benchmark.py > /dev/null (the memory figure was measured separately by comparing process RSS before and after import and after 1,000 calls).

Roadmap

• LiteLLM integration
• LangChain callback handler
• Grafana dashboard templates
• Custom scorer plugin API
• Async evaluation support
• CLI tool for offline batch scoring

Contributing

Contributions welcome. See CONTRIBUTING.md for guidelines.

git clone https://github.com/anilatambharii/argus-ai.git
cd argus-ai
pip install -e ".[dev]"
pytest tests/ -v

About

Built by Anil Prasad at Ambharii Labs.

G-ARVIS framework published in "Field Notes: Production AI" on LinkedIn.

ARGUS: Autonomous Runtime Guardian for Unified Systems.

License

Apache 2.0. See LICENSE.