Skip to main content

Agentic Test Framework

An AI-powered browser testing framework that accepts natural language test descriptions and executes them using Playwright.

Quick Start

# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install the package
pip install agentic-test-framework

# Install Playwright browsers
playwright install

# Set up environment
cp .env.example .env
# Edit .env and add your OpenAI API key

# Run an example test
python -m agentic_test_framework "Go to example.com and take a screenshot"

Features

  • 🤖 Natural Language Tests: Write tests in plain English
  • 🌐 Browser Automation: Powered by Playwright (Chrome, Firefox, Safari)
  • 🧠 AI-Driven: Uses OpenAI to interpret test scenarios
  • 📸 Auto Screenshots: Captures screenshots on demand or for every step
  • 📊 Dual Reporting: Custom HTML reports + Playwright trace viewer
  • 🎬 Time-Travel Debugging: Playwright traces with DOM snapshots, network logs, console output
  • 🎥 Video Recording: Automatic video capture of test execution
  • ✅ Assertions: Verify conditions and extract data
  • 🔄 Smart Retries: Handles flaky elements automatically
  • 📝 ATC Format: Structured test files with YAML-like syntax

Example Tests

Command Line

# Simple navigation
agentic-test "Go to google.com and search for 'playwright testing'"

# Multi-step workflow
agentic-test "Navigate to github.com, click Sign in, type 'testuser' in username field"

# Validation
agentic-test "Go to example.com, verify the page title contains 'Example', take screenshot"

ATC File Format

Create structured test files with .atc extension:

# Login Test Suite

@config browser=chromium
@config headless=false

## Scenario: Successful Login
@tag smoke
@tag login

Go to example.com/login
Type 'testuser' into username field
Type 'password123' into password field
Click login button
Verify page contains 'Welcome'
Take a screenshot

Quick Start: Generate ATC Templates

# Create basic template
agentic-test --create tests/my_test.atc

# Create from predefined templates
agentic-test --create tests/login.atc --template login
agentic-test --create tests/shop.atc --template ecommerce
agentic-test --create tests/api.atc --template api

Available templates:

  • basic - Simple test structure
  • login - Login/authentication flows
  • ecommerce - Shopping and checkout flows
  • api - API/integration tests

Run ATC files:

# Run all scenarios
agentic-test tests/login.atc

# Run specific scenario
agentic-test tests/login.atc --scenario "Successful Login"

# Run by tag
agentic-test tests/login.atc --tag smoke

See docs/ATC_FORMAT.md for complete format specification.

Architecture

Natural Language → OpenAI Parser → Action Objects → Playwright Executor → Results

See .github/copilot-instructions.md for detailed architecture documentation.

Project Structure

agentic-test-framework/
├── agentic_test_framework/
│   ├── actions/          # Action type definitions
│   ├── parser/           # OpenAI integration
│   ├── executor/         # Playwright execution engine
│   ├── runner/           # Test orchestration
│   └── reporter/         # HTML report generation
├── config/               # Configuration files
├── examples/             # Example test scenarios
├── docs/                 # Documentation
└── tests/                # Framework tests

HTML Reports

Every test automatically generates a beautiful HTML report with:

  • ✅ Pass/fail status for each step
  • 📸 Embedded screenshots
  • 📊 Extracted data
  • ⏱️ Execution timing
  • 🎨 Color-coded results

Reports are saved to test-results/report_YYYYMMDD_HHMMSS.html

See docs/HTML_REPORTS.md for details.

Development

Running Unit Tests

The framework includes a comprehensive test suite with pytest:

# Run all tests
pytest

# Run with verbose output
pytest -v

# Run with coverage report
pytest --cov=agentic_test_framework --cov-report=html

# Run specific test file
pytest tests/test_actions.py

# Open coverage report in browser
xdg-open htmlcov/index.html  # Linux
open htmlcov/index.html       # macOS

See tests/README.md for detailed testing documentation.

License

MIT

Metadata

Release files for agentic-test-framework 0.1.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for agentic-test-framework 0.1.3
File Size Uploaded Provenance Actions
agentic_test_framework-0.1.3.tar.gz 31.8 kB Without provenance Details for agentic_test_framework-0.1.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for agentic-test-framework 0.1.3
File Interpreter ABI Platform Actions
agentic_test_framework-0.1.3-py3-none-any.whl Python 3 none any Details for agentic_test_framework-0.1.3-py3-none-any.whl

Total release size: 59.7 kB

Release files / agentic_test_framework-0.1.3.tar.gz

Download URL agentic_test_framework-0.1.3.tar.gz
Size 31.8 kB
Tags Source
SHA-256 checksum
How to use checksums
fd3a5776229b0b3a41b58c8621116bb88609de2581ce462ac601ab71e22f074b
BLAKE2b-256 checksum
How to use checksums
299599595c18fe36ae845ea777c2a88ed4e33f1e8a5d4fa50b5d47c0524628df
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.7

Release files / agentic_test_framework-0.1.3-py3-none-any.whl

Download URL agentic_test_framework-0.1.3-py3-none-any.whl
Size 28.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
10f72f867fdd315f5a86cd671272c5c443bc6691c1c4557ccc1a1180c0b592be
BLAKE2b-256 checksum
How to use checksums
308e00a2e898be38d637f649afa43f625f727b56fe1e234432a55ca54ae9d0be
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.7

Security for agentic-test-framework 0.1.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.3 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page