Agentic Test Framework
An AI-powered browser testing framework that accepts natural language test descriptions and executes them using Playwright.
Quick Start
# Create and activate virtual environment
python3 -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
# Install the package
pip install agentic-test-framework
# Install Playwright browsers
playwright install
# Set up environment
cp .env.example .env
# Edit .env and add your OpenAI API key
# Run an example test
python -m agentic_test_framework "Go to example.com and take a screenshot"
Features
- 🤖 Natural Language Tests: Write tests in plain English
- 🌐 Browser Automation: Powered by Playwright (Chrome, Firefox, Safari)
- 🧠 AI-Driven: Uses OpenAI to interpret test scenarios
- 📸 Auto Screenshots: Captures screenshots on demand or for every step
- 📊 Dual Reporting: Custom HTML reports + Playwright trace viewer
- 🎬 Time-Travel Debugging: Playwright traces with DOM snapshots, network logs, console output
- 🎥 Video Recording: Automatic video capture of test execution
- ✅ Assertions: Verify conditions and extract data
- 🔄 Smart Retries: Handles flaky elements automatically
- 📝 ATC Format: Structured test files with YAML-like syntax
Example Tests
Command Line
# Simple navigation
agentic-test "Go to google.com and search for 'playwright testing'"
# Multi-step workflow
agentic-test "Navigate to github.com, click Sign in, type 'testuser' in username field"
# Validation
agentic-test "Go to example.com, verify the page title contains 'Example', take screenshot"
ATC File Format
Create structured test files with .atc extension:
# Login Test Suite
@config browser=chromium
@config headless=false
## Scenario: Successful Login
@tag smoke
@tag login
Go to example.com/login
Type 'testuser' into username field
Type 'password123' into password field
Click login button
Verify page contains 'Welcome'
Take a screenshot
Quick Start: Generate ATC Templates
# Create basic template
agentic-test --create tests/my_test.atc
# Create from predefined templates
agentic-test --create tests/login.atc --template login
agentic-test --create tests/shop.atc --template ecommerce
agentic-test --create tests/api.atc --template api
Available templates:
- basic - Simple test structure
- login - Login/authentication flows
- ecommerce - Shopping and checkout flows
- api - API/integration tests
Run ATC files:
# Run all scenarios
agentic-test tests/login.atc
# Run specific scenario
agentic-test tests/login.atc --scenario "Successful Login"
# Run by tag
agentic-test tests/login.atc --tag smoke
See docs/ATC_FORMAT.md for complete format specification.
Architecture
Natural Language → OpenAI Parser → Action Objects → Playwright Executor → Results
See .github/copilot-instructions.md for detailed architecture documentation.
Project Structure
agentic-test-framework/
├── agentic_test_framework/
│ ├── actions/ # Action type definitions
│ ├── parser/ # OpenAI integration
│ ├── executor/ # Playwright execution engine
│ ├── runner/ # Test orchestration
│ └── reporter/ # HTML report generation
├── config/ # Configuration files
├── examples/ # Example test scenarios
├── docs/ # Documentation
└── tests/ # Framework tests
HTML Reports
Every test automatically generates a beautiful HTML report with:
- ✅ Pass/fail status for each step
- 📸 Embedded screenshots
- 📊 Extracted data
- ⏱️ Execution timing
- 🎨 Color-coded results
Reports are saved to test-results/report_YYYYMMDD_HHMMSS.html
See docs/HTML_REPORTS.md for details.
Development
Running Unit Tests
The framework includes a comprehensive test suite with pytest:
# Run all tests
pytest
# Run with verbose output
pytest -v
# Run with coverage report
pytest --cov=agentic_test_framework --cov-report=html
# Run specific test file
pytest tests/test_actions.py
# Open coverage report in browser
xdg-open htmlcov/index.html # Linux
open htmlcov/index.html # macOS
See tests/README.md for detailed testing documentation.
License
MIT
Metadata
Release files for agentic-test-framework 0.1.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| agentic_test_framework-0.1.3.tar.gz | 31.8 kB | Without provenance | Details for agentic_test_framework-0.1.3.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| agentic_test_framework-0.1.3-py3-none-any.whl | Python 3 | none | any | Details for agentic_test_framework-0.1.3-py3-none-any.whl |
Total release size: 59.7 kB
Release files / agentic_test_framework-0.1.3.tar.gz
| Download URL | agentic_test_framework-0.1.3.tar.gz |
|---|---|
| Size | 31.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
fd3a5776229b0b3a41b58c8621116bb88609de2581ce462ac601ab71e22f074b
|
|
BLAKE2b-256 checksum How to use checksums |
299599595c18fe36ae845ea777c2a88ed4e33f1e8a5d4fa50b5d47c0524628df
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Release files / agentic_test_framework-0.1.3-py3-none-any.whl
| Download URL | agentic_test_framework-0.1.3-py3-none-any.whl |
|---|---|
| Size | 28.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
10f72f867fdd315f5a86cd671272c5c443bc6691c1c4557ccc1a1180c0b592be
|
|
BLAKE2b-256 checksum How to use checksums |
308e00a2e898be38d637f649afa43f625f727b56fe1e234432a55ca54ae9d0be
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Security for agentic-test-framework 0.1.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.