🧠 COGNIX

A domain-agnostic, uncertainty-aware multi-agent AI research framework.

COGNIX is a framework designed to evaluate decision-making in Multi-Agent AI systems under conditions of distributional shift and sensor degradation. It provides a modular architecture for quantifying uncertainty (Epistemic vs. Aleatoric), fusing multi-agent beliefs, and generating calibrated, risk-aware decisions.

🏗️ 1. Framework Architecture

COGNIX is not a monolithic script, but a decoupled, dependency-injected framework built around strict API contracts. The core DecisionEngine coordinates the pipeline without depending on concrete algorithmic implementations.

The framework is divided into pluggable interfaces:

• AgentInterface: Standardizes agent predictions and uncertainty estimations (PredictionResult and UncertaintyResult).
• GraphRefinement: Handles inter-agent communication and attention routing (e.g., StandardGAT, EpistemicGAT, NoGraph).
• BeliefFuser: Implements belief fusion strategies (e.g., EpistemicWeightedFusion, AverageFusion).
• Calibrator: Enforces post-hoc calibration via Inductive Conformal Prediction.
• AttributionMethod: Explains decision contributions (e.g., EpistemicShapley).
• RiskStrategy: Maps calibrated uncertainties to actionable escalation levels.
• TransportInterface: Minimal abstraction for distributed multi-agent deployment.

🔬 2. Research Mechanism

The central research mechanism evaluated by COGNIX flows as follows:

    epistemic uncertainty
            ↓
    epistemic prior
            ↓
    GAT attention
            ↓
    collective multi-agent belief
            ↓
    conformal calibration
            ↓
    risk-aware decision
            ↓
    attribution / escalation

By explicitly isolating epistemic uncertainty (model ignorance or out-of-distribution shift), COGNIX provides a mechanism to reduce the prior influence of uncertain agents before the network graph routes attention. The specific sender prior evaluated in Experiment 2B showed no descriptive evidence of additional held-out benefit over StandardGAT.

📊 3. Experimental Evidence

COGNIX is evaluated under a rigorous benchmarking infrastructure (Phase B). In the tested synthetic conditions:

• Epistemic vs Standard Attention: Ablation studies provide evidence that injecting an epistemic prior materially changes the attention redistribution of the GAT relative to a standard learned attention mechanism.
• Out-Of-Distribution (OOD) Shift: Observed in the tested conditions, epistemic-weighted fusion reduced the Expected Calibration Error (ECE) compared to uniform weighting when targeted agents were subjected to severe covariate shift.
• Fail-Loud Research Integrity: The pipeline strictly enforces a fail-loud behavior in research mode (e.g., rejecting fallback values when uncertainty decomposition fails), guaranteeing mathematically sound provenance for all reported metrics.

Note: The magnitude of these effects depends heavily on the training quality of the agents and the severity of the distributional shift.

Closed Experiment 2B: official held-out TEST. All 15 frozen runs (NoGraph, StandardGAT, and EpistemicGAT; five paired seeds) were evaluated on 627 scenarios and 187,473 scored timesteps per run. StandardGAT improved BCE relative to NoGraph for every seed, but reduced corruption F1 under the frozen GRAPH_VAL thresholds for every seed. Graph communication's benefit was therefore metric-dependent. EpistemicGAT was approximately neutral to slightly worse than StandardGAT. The prespecified 627-scenario macro-AUROC was undefined because all 107 normal scenarios were single-class; none were dropped. Five-seed comparisons are descriptive, with no prespecified or performed inferential significance test.

q_normal represents P(target = normal) under the TRAIN-derived constructed clean-vs-pseudo task and must not be interpreted as a probability of physical driving safety. p_corruption = 1 - q_normal. These findings concern the evaluated graph sender prior and do not extend the separate synthetic fusion findings above to official TEST. See the comprehensive research report for tables, figures, protocol details, and limitations. Experiment 2B is closed and immutable; its TEST evaluation must not be rerun or retuned.

🎯 4. Research Targets

The ongoing development of COGNIX targets the following scientific objectives:

• Demonstrating behavioral substitutability across multiple non-automotive domains (e.g., tabular classification, distributed sensor networks).
• Expanding the conformal calibration pipeline to map empirical coverage under severe, non-exchangeable distribution shifts.
• Benchmarking the latency and exact vs. approximate runtime of Epistemic Shapley attribution as the number of agents $N$ scales.
• Evaluating resilience against conflicting agents and communication dropouts via the distributed TransportInterface.

⚠️ 5. Limitations

COGNIX is an early-stage research framework. Users must be aware of the following:

• No Physical Safety Guarantees: COGNIX does not guarantee safety. Conformal prediction provides empirical coverage only under strict calibration and exchangeability assumptions, which do not hold universally under arbitrary OOD shift.
• No Superiority Claims: COGNIX does not universally outperform baselines. Its mechanisms are highly dependent on the quality of the underlying MC-Dropout or Deep Ensemble uncertainty estimations.
• Latency: COGNIX makes no claims of "real-time" performance. Online latency for MC-Dropout, GAT inference, and exact Shapley enumeration must be explicitly measured for your specific hardware and batch size constraints.
• Approximate Mathematics: Uncertainty decomposition relies on approximations (e.g., Bernoulli predictive-variance decomposition) that may degrade under certain model architectures.

🚀 6. Example Usage

Installation

Install the generic framework from PyPI:

pip install cognix-uncertainty-framework

To include optional feature extras (such as Graph Attention Networks, explainability, or vision):

pip install "cognix-uncertainty-framework[graph,explainability]"

Or clone and install from the actual source repository:

# Clone the repository
git clone https://github.com/AdityaR4685/Cognix_.git
cd Cognix_

# Install in editable mode (with development dependencies)
pip install -e ".[dev]"

Dependency Injection Pipeline

from cognix.engine.decision_engine import DecisionEngine
from cognix.config.schema import CognixConfig
from cognix.belief.fusion import EpistemicWeightedFusion
from cognix.calibration.conformal import ConformalPredictor

# Initialize base domain-agnostic strategies (pure Python / NumPy)
config = CognixConfig()
fuser = EpistemicWeightedFusion()

# Inject into the DecisionEngine (uses graph="no_graph" by default on base installs)
engine = DecisionEngine(
    config=config,
    belief=fuser,
    graph="no_graph",
    mode="production"  # Use mode="research" for strict mathematical assertion checks
)

# Agents must implement the AgentInterface (or use cognix.agents.adapters)
agents = [radar_agent, camera_agent, lidar_agent]

# Evaluate the pipeline
result = engine.decide(agents, input_data)

print(f"Fused Prediction: {result.confidence}")
print(f"Total Uncertainty: {result.total_uncertainty}")
print(f"Decision Outcome: {result.decision}")

Conformal Calibration (Optional Plug-in)

To add distribution-free uncertainty calibration, provide a fitted ConformalPredictor:

from cognix.calibration.conformal import ConformalPredictor

calibrator = ConformalPredictor()
# Fit on calibration split: probabilities shape (N, C), labels shape (N,)
calibrator.fit(cal_outputs=val_probs, cal_labels=val_labels)

engine = DecisionEngine(
    config=config,
    belief=fuser,
    graph="no_graph",
    calibrator=calibrator,
    mode="research"
)

Graph Attention Networks (Optional Extra)

To utilize relational message passing and Epistemic Graph Attention Networks:

pip install "cognix-uncertainty-framework[graph]"

from cognix.graph.epistemic_gat import EpistemicGAT

gat = EpistemicGAT(input_dim=3, hidden_dim=8, output_dim=1)
engine = DecisionEngine(
    config=config,
    belief=fuser,
    graph=gat,
    calibrator=calibrator,
    mode="research"
)

Running Automated Benchmarks

COGNIX supports automated, multi-seed ablation benchmarking:

python -m cognix.evaluation.run \
    --config configs/benchmark/gat_ablation.yaml \
    --output-dir results/gat_ablation/

CARLA Live Demonstration V2

COGNIX includes a real-time, closed-loop demonstration in the CARLA autonomous driving simulator (cognix/evaluation/cognix_live_demo_v2.py). It connects to a running CARLA simulator (Town10HD_Opt), deploys multi-agent epistemic belief fusion, runs inductive conformal prediction, and performs risk escalation and route recovery under interactive or automated sensor faults.

Launching the Live Demonstration

Ensure CARLA 0.9.15+ is running on the target machine:

# Launch the live demonstration runner
python -m cognix.evaluation.cognix_live_demo_v2 --host 127.0.0.1 --port 2000

# Automated scenario cycling and telemetry capture
python -m cognix.evaluation.cognix_live_demo_v2 --host 127.0.0.1 --port 2000 --cycle-scenarios --duration 120 --output reports/carla_runtime_v1/evaluation_runs/live_agent_uncertainty_demo_001/telemetry_route_efficiency_120s.json

Interactive Controls

Verification Artifacts & Telemetry

Full evaluation reports, multi-sensor telemetry traces, and efficiency audits for the live demonstration are archived in: reports/carla_runtime_v1/evaluation_runs/live_agent_uncertainty_demo_001/ See COGNIX_LIVE_DEMO_V2_ROUTE_EFFICIENCY_REPORT.md for the benchmarked route completion, speed profiles, and conformal set size dynamics.

🤝 Contributing

We welcome contributions! Please see our Contributing Guidelines and Code of Conduct for details on how to get started.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.