UUV Assistant AI based, used to improve the life of testers and developers by generating cucumber phrases from the GUI.

Navigation

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for luifr10 from gravatar.com luifr10

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Louis Fredice NJAKO MOLOM, Stanley SERVICAL
  • Maintainer: Louis Fredice NJAKO MOLOM, Stanley SERVICAL
  • Tags a11y , acceptance , accessibilite , accessibility , ai , artificial-intelligence , bdd , cucumber , cypress , e2e , end 2 end , end2end , gherkin , image-analysis , llm , playwright , tdd , test , test-generation , testing , testing-library , uuv , vlm
  • Requires: Python <3.15, >=3.10
  • Provides-Extra: mlflow
Classifiers
Report project as malware

Project description

uuv-assistant-ai

UUV Logo

Assistant AI - Image-based test generation

AI-powered assistant that helps testers and developers generate Cucumber BDD test scenarios from GUI screenshots and HTML content.

PyPI version python fastapi poetry Support us on Open Collective

What is uuv-assistant-ai?

uuv-assistant-ai is an AI-powered service that extends the UUV ecosystem by enabling image-based test generation. It uses Vision Language Models (VLMs) and Large Language Models (LLMs) to:

  • Describe images - Generate natural language descriptions of UI screenshots
  • Classify images - Determine if UI elements are decorative or functional
  • Generate test scenarios - Convert visual UI analysis into Cucumber BDD test steps

This service integrates with the main @uuv/assistant to provide a complete solution for visual test generation.

Features

  • 🖼️ Image Description - Uses VLMs to describe UI screenshots in natural language
  • 🧠 Image Classification - Classifies images as decorative or functional using LLMs
  • 🔄 Streaming API - Real-time streaming of analysis results
  • 📊 MLflow Integration - Experiment tracking for model evaluation
  • 🤖 DSPy Framework - Built with DSPy for optimized AI workflows
  • 🌐 FastAPI Server - RESTful API for easy integration

Installation

Prerequisites

  • Python >=3.10, <3.15

Setup with pip (Recommended)

pip install uuv-assistant-ai

Setup with uv

# Install the package from PyPI
uv add uuv-assistant-ai

# Or install with MLflow support
uv add uuv-assistant-ai[mlflow]

Environment Variables

Create a .env file with the following variables:

Required (for API access)

LLM_API_URL=https://api.openai.com/v1
LLM_API_KEY=your-openai-api-key
LLM_MODEL=gpt-4

VLM_API_URL=https://api.openai.com/v1
VLM_API_KEY=your-openai-api-key
VLM_MODEL=gpt-4-vision-preview

Optional (for experiment tracking)

MLFLOW_TRACKING_URI=http://localhost:5000

Usage

Start the Server

# After installing from PyPI
uuv-assistant-ai

# Or using uvx (run without installing)
uvx uuv-assistant-ai

# For development (from source)
uv run uuv-assistant-ai

The API will be available at http://localhost:8000

Programmatic Usage

from uuv_assistant_ai.image_classifier import (
    UUVMultipleImageDescriberAgent,
    UUVImageClassifierAgent
)
from PIL import Image

# Describe images
describer = UUVMultipleImageDescriberAgent(
    vlm_api_url="https://api.openai.com/v1",
    vlm_api_key="your-key",
    vlm_model="gpt-4-vision-preview"
)

image = Image.open("screenshot.png")
descriptions = describer(image)

# Classify images
classifier = UUVImageClassifierAgent(
    llm_api_url="https://api.openai.com/v1",
    llm_api_key="your-key",
    llm_model="gpt-4"
)

result = classifier(
    html_content="<html>...</html>",
    css_selector=".element",
    image_description="A button labeled Submit"
)

API Endpoints

1. Classify Image (Unified)

Stream image analysis results including description and classification.

curl -X POST "http://localhost:8000/api/v1/image/classify-unified" \
  -F "html_content=<html>" \
  -F "css_selector=.element-selector" \
  -F "target_img_file=@screenshot.png"

Response (Server-Sent Events):

{"image_description": "A button labeled 'Submit' with blue background"}
{"is_decorative": false, "confidence": 0.95, "analysis_details": "This is a functional button..."}

2. Multiple Image Description

Describe multiple images in a single request.

curl -X POST "http://localhost:8000/api/v1/image/multiple-describe" \
  -F "target_img_file=@screenshot.png"

Response:

{
    "descriptions": [
        { "element": "button", "description": "Submit button" },
        { "element": "input", "description": "Text input field" }
    ]
}

3. Classify Image (Standard)

Classify an image using a pre-computed description, html_content and css_selector.

curl -X POST "http://localhost:8000/api/v1/image/classify" \
  -F "html_content=<html>" \
  -F "css_selector=.element-selector" \
  -F "image_description=A button labeled Submit"

Response:

{
    "is_decorative": false,
    "confidence": 0.95,
    "analysis_details": "This is a functional button..."
}

When to use which endpoint:

Quick Start (Recommended)
Use classify-unified for a simple, one-call approach that returns both image description and classification.

Advanced / Step-by-step
For more control, use the two-step approach:

  1. multiple-describe - Get descriptions for all UI elements in one screenshot
  2. classify - Classify each element using the pre-computed description

This step by approach is useful when you need to:

  • Reuse descriptions for multiple operations
  • Analyze multiple elements separately
  • Build custom workflows with intermediate processing

Development

Installation

# From PyPI
pip install uuv-assistant-ai
# or
uv add uuv-assistant-ai

# From source (for development)
pip install -e .
# or
cd packages/assistant-ai
uv sync

Run Tests

# Using pip
pip install pytest pytest-cov pytest-html pytest-sugar
pytest tests/

# Using uv
uv run pytest tests/

Lint and Format

# Lint
uv run ruff check uuv_assistant_ai tests/

# Format
uv run ruff format uuv_assistant_ai tests/

Build

uv build

Project Structure

uuv_assistant_ai/
├── __init__.py
├── main.py              # FastAPI application
├── server.py            # Server startup
└── image_classifier/
    ├── __init__.py
    ├── agent.py         # Main workflow agent
    ├── model.py         # Data models
    └── sub_agents/
        ├── image_classifier.py   # Image classification agent
        └── image_describer.py    # Image description agent

Integration with UUV Ecosystem

The uuv-assistant-ai service integrates with:

Documentation

Full documentation: https://e2e-test-quest.github.io/uuv/

License


MIT license

This project is licensed under the terms of the MIT license.

Authors

Support UUV

If you want to help UUV grow, you can fund the project directly via Open Collective. Every contribution helps us dedicate more time and energy to improving this open-source tool.

Project details

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for luifr10 from gravatar.com luifr10

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Louis Fredice NJAKO MOLOM, Stanley SERVICAL
  • Maintainer: Louis Fredice NJAKO MOLOM, Stanley SERVICAL
  • Tags a11y , acceptance , accessibilite , accessibility , ai , artificial-intelligence , bdd , cucumber , cypress , e2e , end 2 end , end2end , gherkin , image-analysis , llm , playwright , tdd , test , test-generation , testing , testing-library , uuv , vlm
  • Requires: Python <3.15, >=3.10
  • Provides-Extra: mlflow
Classifiers

Release history Release notifications | RSS feed

1.8.0

1.7.0

1.4.0

1.3.0

1.2.0

1.1.0

1.0.1

This version

1.0.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

uuv_assistant_ai-1.0.0.tar.gz (377.5 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

uuv_assistant_ai-1.0.0-py3-none-any.whl (13.9 kB view details)

Uploaded Python 3

File details

Details for the file uuv_assistant_ai-1.0.0.tar.gz.

File metadata

  • Download URL: uuv_assistant_ai-1.0.0.tar.gz
  • Upload date:
  • Size: 377.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.7

File hashes

Hashes for uuv_assistant_ai-1.0.0.tar.gz
Algorithm Hash digest
SHA256 d9f5c0b4123908bbecbc601a3bcfccfb9f4ad43c71930722da1eb14de5823624
MD5 e3ed9c492723b8017fb98939eb470403
BLAKE2b-256 7990a072438f5d1acc13707c865db76cca830db82aa8bf47c4cb0ef741756a3b

See more details on using hashes here.

Provenance

The following attestation bundles were made for uuv_assistant_ai-1.0.0.tar.gz:

Publisher: uuv-github-ci.yml on e2e-test-quest/uuv

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file uuv_assistant_ai-1.0.0-py3-none-any.whl.

File metadata

File hashes

Hashes for uuv_assistant_ai-1.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 ecf1fd72d8cb558b3888a00abb19a2b8acefe47efc21efda3e8890ac9514f73c
MD5 fb5c822177ea3d54fb1115d3b7551bb2
BLAKE2b-256 4eba621df0ab6de6ded150b1a82c791e2403193d1695fddd6bce9b0219afaff7

See more details on using hashes here.

Provenance

The following attestation bundles were made for uuv_assistant_ai-1.0.0-py3-none-any.whl:

Publisher: uuv-github-ci.yml on e2e-test-quest/uuv

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page