autotester 0.0.7

Autotester - E2E testing for your codebase

Autotester

Autotester is an open-source testing automation tool to make E2E automated.

• 🤖 Run end-to-end tests using natural language descriptions
• 🐛 Detect potential bugs and provide detailed fix explanations
• ⚡ Reduce testing overhead while improving code quality

Currently supporting Python and TypeScript, with more languages coming soon.

Quickstart

Install the package

pip install autotester

Add a config file to your project called autotester.yml. This tells Autotester what to test and how.

e2e:
  login-test: # Name of the test. You can add more
    url: "localhost:3000" # Starting URL of your app. It can be a local server or a remote server
    steps:
      - Login with Github
      - Go to the team page
      - Change the team name to "e2e"
      - Click on the "Save" button
      - Check that the team name is "e2e" # use words like "Check that" to assert the results of the test

If your environment is protected by HTTP Basic Auth, add an auth block:

e2e:
  auth:
    type: basic
    username: "dev"
    password: "dev123"
  login-test:
    url: "https://staging.example.com"
    steps:
      - Check the homepage loads

You can also provide auth credentials via environment variables (these take precedence over YAML):

export AUTOTESTER_AUTH_USERNAME="dev"
export AUTOTESTER_AUTH_PASSWORD="dev123"

That's it. To run it, you need to have an OpenAI API key and Chrome/Chromium installed.

If you don't already have Chrome installed, you can use the browser-use CLI to install Chromium:

browser-use install

Then export your OpenAI key and run:

export OPENAI_API_KEY="your-openai-api-key"
autotester

You will get a summary report like the following:

🖥️ 1/1 E2E tests

login-test: Success!

GitHub Action

Autotester can be used in a GitHub Action to run E2E tests after you release a new version.

Check out the action's README for more information, but here's a quick example:

name: Run Autotester

on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  after-deployment:
    runs-on: ubuntu-latest
    steps:
      - uses: cyberwave-os/autotester-action@v0.1.0
        with:
          action-type: "e2e"
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          STARTING_URL: "http://yourstaging.yourwebsite.com"
          # Optional: for Basic Auth protected environments
          # AUTOTESTER_AUTH_USERNAME: ${{ secrets.AUTOTESTER_AUTH_USERNAME }}
          # AUTOTESTER_AUTH_PASSWORD: ${{ secrets.AUTOTESTER_AUTH_PASSWORD }}

CLI Reference

Commands

• autotester: Without any command, runs E2E tests if defined in autotester.yml
• autotester e2e: Runs end-to-end tests defined in autotester.yml

Command Options

Global Options

• -v, --verbose: Enable verbose logging output
• --version: Display Autotester version number

E2E Test Command

autotester e2e [--config <config_file>] [--verbose]

• --config: (Optional) Path to the YAML configuration file (defaults to autotester.yml)
• -v, --verbose: (Optional) Enable verbose logging output

Environment Variables

• OPENAI_API_KEY: (Required) Your OpenAI API key
• CHROME_INSTANCE_PATH: Path to your Chrome instance. Defaults to /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
• AUTOTESTER_AUTH_USERNAME: Username for HTTP Basic Auth (overrides auth.username in YAML)
• AUTOTESTER_AUTH_PASSWORD: Password for HTTP Basic Auth (overrides auth.password in YAML)

Run Tests with Docker

For contributors who want a reproducible test environment, you can run the test suite in Docker.

From the repository root:

make test-docker

This command mounts your local project into the container and runs pytest. If you want to build manually first:

make test-docker-build
docker compose -f tests/docker-compose.yml run --rm test

Roadmap

• Add support for simple auth, since test environment often have auth.
• Add support for screen recording with Posthog, so that QA engineers can easily review failed tests.

Credits

• Powered by BrowserUse
• This project evolved from an earlier open-source project developed by Tailor Media Inc, relased under Apache 2.0 Codebeaver