aisa-checker 0.1.0

AISA - Agentic AI Systems Architecture Compliance Checker. Evaluate agentic AI projects against the AISA 7-layer reference architecture.

AISA - Agentic AI Systems Architecture Compliance Checker

Evaluate your agentic AI system against the AISA reference architecture.

AISA checks your project across 7 architectural layers and 4 cross-layer contracts, identifying gaps in governance, evaluation, safety, and operational readiness that most agentic AI frameworks miss.

Quick Start

pip install aisa

Pattern-Based Scan (fast, no API key)

aisa check ./my-agent-project

AI-Powered Analysis (deep, requires API key)

# Option 1: Pass API key directly
aisa ai-check ./my-agent-project --api-key sk-...

# Option 2: Use environment variable
export OPENAI_API_KEY=sk-...    # or ANTHROPIC_API_KEY or GOOGLE_API_KEY
aisa ai-check ./my-agent-project

# Option 3: Choose provider
aisa ai-check ./my-agent-project --provider anthropic --api-key sk-ant-...

Full Analysis (best accuracy - combines both)

aisa full-check ./my-agent-project --report

Interactive Assessment (works for any system)

aisa assess

What It Evaluates

AISA evaluates your project across the complete agentic AI lifecycle:

Layer	What's Checked
L1: LLM Foundation	Model adapters, prompt templates, context management, safety filters
L2: Tool & Environment	Tool schemas, validation, sandboxing, permissions, MCP, error handling
L3: Cognitive Agent	Planning, memory, goal tracking, reflection, feedback integration
L4: Infrastructure	Orchestration, state management, multi-agent, tracing, logging
L5: Evaluation	Component tests, trajectory eval, behavioral monitoring, regression tests
L6: Dev & Deployment	Versioning, CI/CD, A/B testing, benchmarking, staged rollout
L7: Governance	Policy-as-code, privacy, fairness, accountability, human oversight

Plus 4 cross-layer contracts: Policy, Telemetry, Versioning, Budget.

Scoring Modes

Mode	Speed	Accuracy	API Key
check	Fast	Good	No
ai-check	Slow	Better	Yes
full-check	Slowest	Best	Yes
assess	Manual	Depends on answers	No

How Scoring Works

• Pattern scan (check): Searches for 200+ code signatures (imports, function calls, config patterns) mapped to AISA criteria. Fast and deterministic.
• AI analysis (ai-check): Sends your code to an LLM that semantically analyzes architecture, not just keywords. Catches patterns that regex misses.
• Combined (full-check): Runs both, takes the higher score per criterion. Most accurate.
• Assessment (assess): 55 yes/no/partial questions. Works for any system, including non-Python or proprietary.

Output

Terminal Report

  LAYER COVERAGE:
  --------------------------------------------------------
  L1 LLM Foundation         ██████░░░░░░░░░░░░░░  33.3%  [Minimal]
  L2 Tool & Environment     ██░░░░░░░░░░░░░░░░░░  14.3%  [Absent]
  L3 Cognitive Agent        ███░░░░░░░░░░░░░░░░░  16.7%  [Absent]
  L4 Infrastructure         ░░░░░░░░░░░░░░░░░░░░   0.0%  [Absent]
  L5 Evaluation             ░░░░░░░░░░░░░░░░░░░░   0.0%  [Absent]
  L6 Dev & Deployment       ██░░░░░░░░░░░░░░░░░░  14.3%  [Absent]
  L7 Governance             ░░░░░░░░░░░░░░░░░░░░   0.0%  [Absent]

  OVERALL AISA SCORE: 9/100
  GRADE: F - Significant Gaps

Markdown Report (--report)

Generates a detailed aisa_report.md with:

• Per-criterion scores with evidence
• Prioritized gap list
• Actionable recommendations per gap
• Framework detection results

Supported Frameworks

Auto-detects and evaluates projects using:

• LangChain / LangGraph
• CrewAI
• Microsoft AutoGen
• OpenAI Assistants / Agents SDK
• Anthropic Claude SDK
• LlamaIndex
• Microsoft Semantic Kernel
• Google ADK

LLM Providers (for ai-check / full-check)

Provider	Environment Variable	Default Model
OpenAI	OPENAI_API_KEY	gpt-4o
Anthropic	ANTHROPIC_API_KEY	claude-sonnet-4-20250514
Google	GOOGLE_API_KEY	gemini-2.0-flash

# Override provider and model
aisa ai-check ./project --provider anthropic --model claude-sonnet-4-20250514

Grading Scale

Grade	Score	Meaning
A	80-100	Production Ready
B	60-79	Maturing
C	40-59	Developing
D	20-39	Early Stage
F	0-19	Significant Gaps

About AISA

AISA (Agentic AI Systems Architecture) is a unified layered reference architecture for designing, deploying, evaluating, and governing agentic AI systems. Proposed by Tuwaiq Academy (TRDC), it provides a shared vocabulary and design blueprint for building production-grade autonomous AI systems.

Paper: Nacar, O., Alquffari, D., & Alkhalifa, M. (2026). "AISA: A Unified Architecture for Agentic AI Systems."

Resources: HuggingFace | Paper PDF

Contributing

Contributions welcome! Areas we'd love help with:

• Additional detection patterns for frameworks
• New layer criteria based on real-world deployments
• Integration with CI/CD platforms
• Support for additional LLM providers

License

MIT License - see LICENSE for details.

Citation

@misc{nacar2026aisa,
  title={AISA: A Unified Architecture for Agentic AI Systems},
  author={Nacar, Omer and Alquffari, Deema and Alkhalifa, Mohammed},
  year={2026},
  institution={TRDC - Tuwaiq Academy},
}