scriptgini 1.7.3

Agentic AI system that converts functional test cases into automation test scripts.

ScriptGini

Enterprise-grade Agentic AI system that converts functional test cases into high-quality, review-ready automation test scripts.

Current release: v1.7.3 (Patch release - Includes security scan orchestration and consolidated reporting automation updates)

Latest validated status (2026-05-12):

• Test run: 286 passed, 4 deselected
• Coverage: 100% on app/ (2952/2952 statements)
• PostgreSQL integration suite: opt-in and skip-by-default unless explicitly enabled

---

What is ScriptGini?

ScriptGini is an AI-powered test automation engine built for Quality Engineering teams. You feed it a functional test case and an Application Under Test (AUT) URL — it returns a production-ready automation script in your chosen framework, generated by a multi-step LangGraph agent that reasons about test intent before writing a single line of code.

---

Features

• Agentic 3-node LangGraph pipeline — Intent analysis → Script generation → Quality review
• Multi-provider LLM support — OpenAI, Ollama (local), OpenRouter, Google Gemini, AWS Bedrock
• Framework-agnostic output — Playwright Python, Selenium Python, UFT VBScript, Cypress JS
• Intelligent selector strategy — Role → Label → data-testid → CSS → XPath (last resort)
• Project & AUT management — Store multiple projects, each with its own base URL and defaults
• Full test case history — Every generated script is stored in SQLite with status and token usage
• Execution history persistence — Every run is stored in script_runs with stdout/stderr, exit code, and duration
• Durable execution APIs — Async execution jobs with run/status/abort endpoints and idempotency-key support
• Hardened execution sandbox — Script runs use isolated Python mode, static safety validation, and restricted environment variables
• Bulk job orchestration — Project-level bulk generate and bulk run with pollable job status
• Run analytics dashboard — Project-level pass/fail/timeout metrics and recent failure feed
• Coverage analytics — Module-level coverage and execution visibility for test cases
• Richer test case intake — Import .txt, .md, .json, .csv, .feature, .yml/.yaml, and .xlsx
• Import preview mapping — Preview parsed scenarios in the UI before creating a project workspace
• Script lifecycle tracking — Refactor, version history, diff, and rollback support
• REST API — FastAPI with auto-generated Swagger UI
• Alembic migrations — Safe, versioned schema management over SQLite

---

Tech Stack

Layer	Technology
API	FastAPI + Uvicorn
Agentic AI	LangGraph + LangChain
LLM Providers	OpenAI, Ollama, OpenRouter, Gemini, Bedrock
Database	SQLite
ORM	SQLAlchemy 2.0
Migrations	Alembic
Config	Pydantic Settings (.env)

---

Quick Start

Windows

start.bat

Linux / macOS

chmod +x start.sh
./start.sh

The script will:

1. Create a Python virtual environment
2. Install all dependencies
3. Copy .env.example → .env if missing (edit it before re-running)
4. Run Alembic migrations
5. Start the server and open Swagger UI in your browser

To load a ready-made sample workspace, use the Load Demo Project button in the web UI or call POST /api/v1/demo/load.

---

Configuration

Copy .env.example to .env and fill in the values you need:

cp .env.example .env

# Choose your default provider
DEFAULT_LLM_PROVIDER=openrouter   # openai | ollama | openrouter | gemini | bedrock

# OpenAI
OPENAI_API_KEY=your_openai_api_key_here

# Ollama (local — no key needed)
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_MODEL=llama3
OLLAMA_NUM_PREDICT=700

# Generation latency controls
LLM_REQUEST_TIMEOUT_SECONDS=45
SCRIPT_GENERATION_TIMEOUT_SECONDS=180
SKIP_REVIEW_FOR_OLLAMA=true

# OpenRouter
OPENROUTER_API_KEY=your_openrouter_api_key_here
OPENROUTER_MODEL=openai/gpt-4o

# Google Gemini
GOOGLE_API_KEY=your_google_api_key_here

# AWS Bedrock
AWS_ACCESS_KEY_ID=your_aws_access_key_id
AWS_SECRET_ACCESS_KEY=your_aws_secret_access_key
AWS_REGION_NAME=us-east-1

.env is git-ignored. Never commit real API keys.

If local generation feels slow, reduce OLLAMA_NUM_PREDICT, keep SKIP_REVIEW_FOR_OLLAMA=true, or switch to a smaller/faster Ollama model.

---

OpenAPI Specification

ScriptGini implements a production-ready OpenAPI 3.0.3 specification that defines a complete enterprise REST API with 50+ endpoints, 60+ schemas, and comprehensive documentation.

Specification Highlights

The API specification includes 12 salient features:

1. Multi-tenant Architecture — Organizations, Workspaces, Teams with hierarchical RBAC
2. Advanced LLM Management — Multi-provider orchestration, health monitoring, cost tracking, and model governance
3. Intelligent Test Data Management — Multi-source ingestion, synthetic generation, PII masking, state locking, placeholder mapping
4. Asynchronous Job Management — HTTP 202 Accepted, idempotent execution, job polling, webhooks
5. Comprehensive Reporting — Structured logs, artifact storage, signed downloads, execution diagnostics
6. Analytics & Insights — Dashboards, trend analysis, flakiness detection, code coverage, LLM usage
7. Defect Lifecycle — Auto-detection, Jira/Azure/GitHub sync, severity tracking, traceability
8. Script Versioning — Git-style history, quality metrics, refactoring, diff metadata
9. Webhooks — Event-driven notifications, delivery guarantees, retry policies
10. Security & Authorization — JWT + OAuth2, API keys with scopes, RBAC enforcement, rate limiting
11. Error Handling & Observability — Standardized error envelope, request ID tracing, correlation IDs
12. Pagination & Filtering — limit/offset/sort, comprehensive filtering, faceted search

API Reference

Once running, visit:

URL	Description
http://localhost:8000/docs	Swagger UI (interactive)
http://localhost:8000/redoc	ReDoc
http://localhost:8000/health	Health check
http://localhost:8000/openapi.json	Raw OpenAPI 3.0.3 specification

Full OpenAPI Documentation

• Complete Specification: docs/openapi-improved-draft-2026-05-10.yaml
• Feature Documentation: docs/OPENAPI-SPECIFICATION.md — Covers all 12 salient features with examples and flows
• API Architecture Review: docs/api-architecture-review-2026-05-10.md — Analysis of current state vs. target enterprise API

API Domains (50+ Endpoints)

Domain	Endpoints	Purpose
Authentication & IAM	8	Login, token refresh, user management, API keys
Organizations & Teams	6	Multi-tenancy, team management, RBAC
Workspaces & Projects	5	Organizational hierarchy, configuration
Test Data Management	11	Data sets, reservations, masking, synthetic generation
LLM Management	5	Provider registration, model governance, cost tracking
Script Engineering	7	Generation, versioning, quality, refactoring
Test Cases	4	CRUD operations for test definitions
Test Orchestration	5	Execution, cancellation, job management
Execution Reporting	4	Reports, logs, artifacts, diagnostics
Analytics & Insights	5	Dashboards, trends, flakiness, coverage, LLM costs
Defect Management	6	Create, update, link, sync to Jira/Azure/GitHub
Webhooks	5	Event subscriptions, delivery management
Database Admin	1	Alembic migrations (admin only, 2FA required)

Quick API Examples

Core Workflow

1. Create a Project (AUT)

POST /api/v1/projects/

{
  "name": "My Web App",
  "aut_base_url": "https://example.com",
  "default_framework": "playwright_python",
  "selector_preference": "role",
  "auth_hints": "Login with admin/admin on /login"
}

2. Add a Test Case

POST /api/v1/projects/{project_id}/test-cases/

{
  "title": "TC-001 Successful Login",
  "format": "step_based",
  "content": "Step 1: Navigate to /login\nStep 2: Enter username 'admin'\nStep 3: Enter password 'admin123'\nStep 4: Click Login button\nExpected: User is redirected to /dashboard and sees 'Welcome' message",
  "preconditions": "User account exists in the system",
  "test_data_hints": "username=admin, password=admin123"
}

3. Generate a Script

POST /api/v1/projects/{project_id}/test-cases/{tc_id}/scripts/generate

{
  "llm_provider": "openrouter",
  "llm_model": "openai/gpt-4o",
  "framework": "playwright_python"
}

Returns 202 Accepted immediately. The agent runs in the background.

4. Poll for the Result

GET /api/v1/projects/{project_id}/test-cases/{tc_id}/scripts/{script_id}

Status values: pending → generating → completed | failed

5. Run a Generated Playwright Script

POST /api/v1/projects/{project_id}/test-cases/{tc_id}/scripts/{script_id}/run

Returns a persisted run record with:

• status (completed | failed | timed_out)
• stdout, stderr
• exit_code, duration_seconds

Execution safeguards:

• Script content is statically validated before execution.
• Unsafe imports and unsafe builtin calls are rejected and persisted as failed runs.
• Runtime uses Python isolated mode with a restricted environment.

6. List Script Run History

GET /api/v1/projects/{project_id}/test-cases/{tc_id}/scripts/{script_id}/runs

7. Bulk Generate Scripts (Project-level)

POST /api/v1/projects/{project_id}/scripts/bulk-generate

{
  "llm_provider": "openrouter",
  "llm_model": "openai/gpt-4o",
  "framework": "playwright_python",
  "test_case_ids": [1, 2, 3]
}

8. Bulk Run Latest Completed Scripts

POST /api/v1/projects/{project_id}/scripts/bulk-run

9. Poll Bulk Job Status

GET /api/v1/projects/{project_id}/scripts/bulk-jobs/{job_id}

10. Get Run Analytics (Project-level)

GET /api/v1/projects/{project_id}/analytics/runs

Returns aggregate execution metrics and latest failure details.

---

LangGraph Agent Pipeline

┌─────────────────┐     ┌──────────────────┐     ┌───────────────┐
│  parse_intent   │────▶│ generate_script  │────▶│ review_script │
│                 │     │                  │     │               │
│ Extracts:       │     │ Produces full    │     │ QA checks:    │
│ • Business goal │     │ framework-       │     │ • Assertions  │
│ • Actions list  │     │ specific script  │     │ • TODO markers│
│ • Assertions    │     │                  │     │ • Rewrites if │
│ • Preconditions │     │                  │     │   needed      │
└─────────────────┘     └──────────────────┘     └───────────────┘

---

Supported Frameworks

Key	Framework
playwright_python	Playwright for Python (default)
selenium_python	Selenium WebDriver Python
uft_vbscript	UFT / QTP VBScript
cypress_js	Cypress JavaScript

---

Project Structure

scriptgini/
├── app/
│   ├── main.py                   # FastAPI application
│   ├── config.py                 # Settings loaded from .env
│   ├── database.py               # SQLAlchemy engine + session
│   ├── models/
│   │   ├── project.py            # Project / AUT model
│   │   ├── test_case.py          # Test case model
│   │   └── generated_script.py  # Script history model
│   ├── schemas/                  # Pydantic request/response schemas
│   ├── routers/
│   │   ├── projects.py           # CRUD — projects
│   │   ├── test_cases.py         # CRUD — test cases
│   │   └── scripts.py            # Generate + history
│   ├── agents/
│   │   ├── script_gini_agent.py  # LangGraph graph definition
│   │   └── prompts.py            # All prompt templates
│   └── llm/
│       └── provider.py           # LLM provider factory
├── alembic/                      # Database migration scripts
├── alembic.ini
├── requirements.txt
├── .env.example                  # Template — copy to .env
├── start.bat                     # Windows launcher
└── start.sh                      # Linux / macOS launcher

---

Database Migrations

Migrations are handled automatically by start.bat / start.sh.

To run manually:

# Apply all pending migrations
alembic upgrade head

# Create a new migration after model changes
alembic revision --autogenerate -m "description"

# Rollback one step
alembic downgrade -1

---

Quality Gate Policy

Every check-in is expected to pass:

1. Unit tests
2. 100% coverage on app/
3. pip-audit
4. Trivy filesystem scan

Local commands:

test.bat
audit.bat
trivy.bat

Opt-in PostgreSQL integration round-trip tests:

set RUN_POSTGRES_INTEGRATION_TESTS=1
set INTEGRATION_DATABASE_URL=postgresql://scriptgini:scriptgini@localhost:5433/scriptgini_test_db
test_integration_postgres.bat

./test.sh
./audit.sh
./trivy.sh

export RUN_POSTGRES_INTEGRATION_TESTS=1
export INTEGRATION_DATABASE_URL=postgresql://scriptgini:scriptgini@localhost:5433/scriptgini_test_db
./test_integration_postgres.sh

Note: Integration tests are intentionally skip-by-default and include a safety check to avoid running against non-test databases.

A CI gate is configured in .github/workflows/quality-gate.yml to enforce the same checks on push/PR.

---

Adding a New LLM Provider

1. Add config keys to app/config.py
2. Add a new _provider() function in app/llm/provider.py
3. Register it in get_llm() and the LLMProvider type alias
4. Add the corresponding key to .env.example

---

Security Notes

ScriptGini ships with a layered security model that is active in the default configuration:

Authentication & Authorization

• JWT Bearer tokens (access + refresh, configurable expiry) protect all non-public endpoints
• API keys (hashed storage, named scopes, revocable) enable CI/CD pipelines without sharing user credentials
• RBAC — four project roles (owner / admin / member / viewer) control read and write access per project
• All auth events (login, register, API-key create/revoke, forbidden access) are written to structured security audit logs

Network Controls

• CORS — only origins listed in CORS_ALLOWED_ORIGINS (env var) are permitted; wildcard * is never set in production configs
• Rate limiting — two layers:
  • Global middleware: 120 req / 60 s per IP for all non-exempt paths (configurable)
  • Per-user/API-key: tighter limits on sensitive actions (login: 10/min, register: 5/min, execution: 20/min, bulk jobs: 10/min)
  • Standard X-RateLimit-* and Retry-After headers are included on all responses

Execution Isolation

• Scripts run inside a constrained runner — only an explicit allowlist of safe import roots is permitted
• Runtime attribute calls are blocked (system, popen, connect, etc.)
• __builtins__, __globals__, and introspection symbols are blocked by name
• Execution environment variables are narrowed to an explicit EXECUTION_ENV_ALLOWED_KEYS allowlist
• Hard + soft execution timeouts enforce maximum run duration

Secrets & Config

• .env is git-ignored — never commit API keys or JWT_SECRET_KEY
• All secrets are consumed via pydantic-settings env-var injection; no hardcoded defaults are safe for production
• JWT_SECRET_KEY must be replaced with a cryptographically random value before deployment
• The UI performs client-side validation only; the agent never makes unsolicited live requests to the AUT

Change Reports

• Commit-range report (dfdf693b..15328b7): docs/changes-dfdf693b-to-15328b7.md

---

Development Roadmap

The project follows an enterprise-grade development roadmap with 6 sprints covering 190 story points over ~12 weeks. See docs/todo.md for detailed sprint breakdown with features, user stories, and tasks.

Sprint Summary

Sprint	Focus	Effort	Status
Sprint 1	IAM Core	30-36pts	🟡 Core delivered (auth hardening pending)
Sprint 2	RBAC + Multi-Tenancy	32-38pts	🟡 Core delivered (RBAC hardening pending)
Sprint 3	Durable Execution	34-40pts	✅ Completed (Redis + Celery queue foundation)
Sprint 4	Security & Hardening	30-36pts	✅ Hardening increment completed (isolation + negative tests + audit controls)
Sprint 5	Reporting & Analytics	28-34pts	✅ Completed (Reports APIs, trends/flakiness, retention cleanup)
Sprint 6	Coverage Analytics, Script Lifecycle, and DX	24-30pts	✅ Completed (coverage analytics, refactor/version history/diff/rollback, 100% quality gate)
Sprint 7	Hardening Completion + Developer Experience	20-24pts	✅ Completed (per-user rate limiting, analytics indexes, OpenAPI scope docs, 286 passed with integration tests deselected by default, 100% coverage)

---

License

MIT