universal-test-framework 1.0.0

Universal polyglot test generation framework with 8-section contract enforcement

Universal Test Framework (UTF)

Polyglot, requirements-driven test generation with hard contract enforcement — via MCP, VS Code, CLI, or Python SDK.

UTF is an MCP server and test-generation engine that enforces an 8-section test contract on every test it produces, regardless of language, project, or prompt quality. Tests that fail the contract are blocked — never silently included — so every test that reaches your suite is traceable, meaningful, and gap-honest.

---

Key Features

• 12 MCP tools — generate, validate, analyze, trace, execute, mutate, report, and query tests
• 6 languages — Python, TypeScript, JavaScript, Java, Go, C++
• 7 test types — unit, integration, API, E2E, contract, performance, security
• 8-section contract — every test must pass all 8 sections or it is blocked (minimum score: 0.85)
• Three install modes — uvx (zero-clone), local venv, or Docker
• VS Code @utf — chat participant with slash commands
• CI/CD ready — GitHub Actions, GitLab CI, pre-commit hooks
• Per-project overrides — YAML rules in .utf/rules/ override global defaults

---

Quick Start

Option 1 — Zero Install (uvx, recommended)

Requires uv. No cloning, no venv management.

Add to %APPDATA%\Code\User\mcp.json (Windows) or ~/.config/Code/User/mcp.json (macOS/Linux):

{
  "servers": {
    "utf": {
      "type": "stdio",
      "description": "Universal Test Framework — 8-section contract enforcement",
      "command": "uvx",
      "args": ["--from", "universal-test-framework", "utf-server", "--transport", "stdio"],
      "cwd": "${workspaceFolder}"
    }
  }
}

Option 2 — Local Clone (VS Code + Copilot)

git clone https://github.com/phoenice-labs/Universal-Test-Framework
cd Universal-Test-Framework
.\scripts\install-vscode-mcp.ps1   # registers MCP server, installs @utf extension, copies prompts

Reload VS Code → open Copilot Chat → type @utf /generate-tests unit.

Option 3 — Docker (Team / Remote)

docker compose up -d utf-server
# MCP server available at http://localhost:8765/sse

---

MCP Tools Reference

Tool	Description
generate_tests	Generate a complete test suite satisfying the 8-section contract
validate_test_contract	Validate any test (generated or hand-written) against the contract
analyze_coverage	Identify coverage gaps in an existing test suite
build_traceability_matrix	Build a requirements → tests traceability matrix
suggest_test_types	Recommend test types with rationale from source code
detect_language_framework	Auto-detect programming language and test framework
query_registry	Query the persistent per-project test registry
run_tests	Execute test files and return structured CI results
run_mutation_tests	Run mutation testing and return mutation score
feedback_status	Get gap analysis, coverage health, and trend report
generate_report	Generate HTML / JUnit / JSON contract compliance report
health	Server health check: version, uptime, tool count

---

The 8-Section Test Contract

Every test generated by UTF must satisfy all 8 sections. Tests that fail any hard section are blocked — returned in blocked_tests, never silently included.

#	Section	What It Must Contain	Weight
1	test_id	Unique ID: TC-{TYPE}-{NNN} (e.g. TC-US-042)	10%
2	why_generated	Rationale tied to a requirement (≥50 chars)	10%
3	requirement_mapping	At least one US-, AC-, REQ-, JIRA-, BUG-, or NFR- reference	15%
4	how_it_exercises	GIVEN / WHEN / THEN with inputs, mocks, assertions (≥100 chars)	20%
5	coverage_contribution	Coverage type + module + estimated %	15%
6	expected_outcome	Precise return values, status codes, state changes (≥50 chars)	15%
7	gaps_missing	Honest list of what this test does NOT cover (≥40 chars)	10%
8	meaningfulness_check	Self-assessment: meaningful / redundant / hallucinated (≥50 chars)	5%

Minimum passing score: 0.85. Tests below this threshold are blocked regardless of individual section presence. "none" in gaps or vague rationale like "to test the function" are rejected.

---

Supported Matrix

Language	Frameworks	Test Types
Python	pytest	unit, integration, api, e2e, security, performance
TypeScript	Jest, Vitest, Playwright	unit, integration, e2e, api
JavaScript	Jest, Vitest	unit, integration
Java	JUnit 5 + AssertJ, Maven	unit, integration, api
Go	go-test + testify	unit, integration, api
C++	Google Test	unit

---

VS Code Integration

After running .\scripts\install-vscode-mcp.ps1 (Option 2) or adding the uvx MCP entry (Option 1):

Copilot Chat Slash Commands

Command	Effect
@utf /generate-tests unit	Unit tests for selected/active code
@utf /generate-tests api	API tests
@utf /generate-tests integration	Integration tests
@utf /generate-tests e2e	End-to-end tests
@utf /generate-tests security	Security / auth tests
@utf /generate-tests performance	Performance / load tests
@utf /validate-contract	Validate a test against the 8-section contract
@utf /coverage-gaps	Identify coverage gaps in the current suite
@utf /traceability	Build requirements → tests traceability matrix
@utf /report	Generate HTML contract compliance report
@utf /status	Server health and installation status

Minimal Prompt Example

You do not need to provide detailed requirements. UTF infers them:

@utf /generate-tests api

UTF reads the open file, detects language and framework, infers requirements from function signatures, generates happy-path + negative + edge-case tests — all validated against the 8-section contract.

---

Python SDK

Use UTF directly in scripts or CI pipelines without the MCP server:

from mcp_server.tools.generate_tests import generate_tests
from mcp_server.tools.validate_contract import validate_test_contract

# Generate tests — UTF infers language, framework, and requirements
result = generate_tests(
    test_type="unit",
    source_code=open("src/auth.py").read(),
    requirements_text="US-101: Users must be authenticated before accessing dashboard",
)

print(result["suite_code"])          # Executable test file
print(result["traceability_matrix"]) # Requirements → tests mapping
print(result["gaps"])                # Identified coverage gaps

# Validate any existing test
validation = validate_test_contract(test_content=my_test_markdown)
print(f"Score: {validation['score']:.0%}  Valid: {validation['is_valid']}")

---

CI/CD Integration

GitHub Actions

# .github/workflows/test-quality-gate.yml
name: UTF Contract Gate
on: [push, pull_request]
jobs:
  contract-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }
      - run: pip install universal-test-framework
      - name: Validate test contract
        run: |
          python - <<'EOF'
          from mcp_server.tools.validate_contract import validate_test_contract
          import glob, sys, pathlib

          failures = []
          for f in glob.glob("tests/**/*.py", recursive=True):
              content = pathlib.Path(f).read_text()
              result = validate_test_contract(test_content=content)
              if not result["is_valid"]:
                  failures.append(f"{f}: score {result['score']:.0%}")
          if failures:
              print("Contract failures:\n" + "\n".join(failures))
              sys.exit(1)
          print("All tests passed contract validation")
          EOF

---

Per-Project Configuration

Create .utf/rules/project-overrides.yaml in your project root:

# Raise minimum score for safety-critical code
contract:
  min_score: 0.90          # default: 0.85
  hard_block_below: 0.75

# Match your Jira project key
traceability:
  requirement_id_pattern: "^(PROJ-\\d+|AC-\\d+(\\.\\d+)?|NFR-\\d+)$"

# Tests generated per requirement per type
scenario_counts:
  happy_path: 1
  negative: 2
  boundary: 1
  edge_case: 1

# Coverage advisory thresholds (appear in report, do not block)
coverage:
  line_target: 85
  branch_target: 75
  mutation_score_target: 70

Rules are deep-merged: project overrides layer on top of UTF's global defaults. The 8-section contract structure itself cannot be overridden.

---

Architecture

UTF MCP Server (stdio or HTTP/SSE)
│
├── mcp_server/server.py         — FastMCP entrypoint, 12 registered tools
│
├── mcp_server/engine/           — Core processing
│   ├── language_detector.py     — Detects language + framework from code/path
│   ├── rule_engine.py           — Loads and merges YAML rules
│   ├── contract_validator.py    — Enforces 8-section contract, scores tests
│   ├── template_renderer.py     — Jinja2 test file generation
│   ├── context_resolver.py      — Resolves project context for generation
│   └── framework_mapper.py      — Maps language → framework → test runner
│
├── mcp_server/tools/            — One module per MCP tool
├── mcp_server/registry/         — SQLite per-project test registry
├── mcp_server/execution/        — pytest, Vitest, Maven, go-test adapters
├── mcp_server/mutation/         — mutmut, Stryker, PIT, Gremlins adapters
├── mcp_server/reporting/        — HTML, JUnit XML, JSON report generation
├── mcp_server/feedback/         — CI listener, trend analysis, gap reopener
│
├── rules/                       — Global YAML rules (language, framework, type)
├── templates/                   — Jinja2 test templates per language/framework
├── agent-customization/         — VS Code copilot-instructions + prompt palette
└── vscode-extension/            — @utf VS Code Chat Participant (VSIX)

Transport modes:

• stdio — default, used by VS Code MCP client and uvx
• HTTP/SSE — for remote/team deployment (--transport http --port 8765)

SQLite registry resolves to .utf/utf.db relative to cwd (the open workspace folder). Each project has its own isolated registry — no shared state.

---

Installation Options Summary

Mode	Command	Requirements
uvx (zero-clone)	uvx --from universal-test-framework utf-server	uv
pip	pip install universal-test-framework	Python 3.11+
Local clone	.\scripts\install-vscode-mcp.ps1	Git, Python 3.11+
Docker	docker compose up -d utf-server	Docker

---

Security

• All MCP tool inputs are validated via Pydantic before processing
• No secrets, credentials, or PII are logged or stored
• The registry (utf.db) is local to each project and never transmitted
• Docker image runs as non-root user
• Rate limiting is enforced on the HTTP/SSE transport

---

License

MIT — see LICENSE