Bubblegum

AI-powered recovery & natural-language execution for Playwright and Appium tests

Bubblegum is NOT a replacement for Playwright or Appium.
It is an intelligent layer that attaches to your existing test framework, heals broken locators using a deterministic-first strategy, and lets you write new steps in plain English.

📘 New here? Start with the focused how‑to guides: How to Use — Web · How to Use — Mobile · How to Use — TypeScript/JS. The User Guide is the single combined reference, and Distribution (PyPI + npm) explains how TS/JS Playwright projects consume Bubblegum.

✨ New in 0.0.6a51 — a reworked AI grounding layer (real AI fallback with a learned cache, semantic matching, tiered model routing, self-hosted screenshot grounding, streaming observability, and a zero-cost replay mode for CI). Everything is opt-in. See AI Grounding Enhancements — setup & real-app testing.

Table of Contents

• What is Bubblegum?
• Quick Start
• Public API
• How It Works
• Resolver Chain
• Configuration
• Error Handling
• Privacy & Security
• Roadmap
• Benchmark Strategy
• Contributing Custom Resolvers
• Architecture
• Project Status

What is Bubblegum?

QA automation breaks when UIs change. Locators go stale, button labels get renamed, and your CI pipeline turns red — not because the feature is broken, but because #login-btn became .btn-login.

Bubblegum solves this with a tiered resolver chain:

1. Deterministic first — tries stable, fast, zero-cost resolvers (accessibility tree, exact text, memory cache)
2. Fuzzy fallback — edit-distance and semantic matching when labels drift slightly
3. AI when needed — LLM/vision models only when deterministic approaches fail

Every decision produces a trace artifact with confidence scores, so your team always knows why a step was resolved the way it was.

Target users (v1): SDETs, automation engineers, framework owners, QA teams already using Playwright or Appium.

Quick Start

Install

# Base package
pip install bubblegum-ai

# Local development install with web extra (Playwright)
pip install -e ".[web]"
python -m playwright install chromium

# Local development install with mobile extra (Appium client)
pip install -e ".[mobile]"
# plus: running Appium server + running emulator/device + installed target app

# Local development install with all optional extras
pip install -e ".[all]"

# Package-user installs (non-editable)
pip install "bubblegum-ai[web]"
pip install "bubblegum-ai[mobile]"

Public import pattern

from bubblegum import act, verify, recover, extract, configure_runtime

Add configuration

Create bubblegum.yaml in your project root:

grounding:
  accept_threshold:  0.85
  review_threshold:  0.70
  ambiguous_gap:     0.05
  reject_threshold:  0.50
  max_cost_level:    medium   # low | medium | high
  enable_vision:     false
  enable_ocr:        true
  memory_ttl_days:   7

ai:
  enabled:   true
  provider:  anthropic          # anthropic | openai | gemini | local
  model:     <your-model-name>  # must be set explicitly — no default

privacy:
  redact_pii:         true
  send_screenshots:   false
  log_provider_calls: true
  process_screenshots_for_vision: false
  process_screenshots_for_ocr:    false

No default model is set intentionally. You must configure both provider and model to avoid surprise API costs.

Recover a failing step in an existing Playwright test

from bubblegum import recover

result = await recover(
    page=page,
    failed_selector="#login-btn",
    intent="Click Login"
)

assert result.status in ["passed", "recovered"]
# result.status == "recovered" means the original selector failed,
# but Bubblegum found and clicked the right element.

That's it. You get value from Bubblegum before writing a single new natural-language test.

Pytest plugin reporting

Bubblegum ships with a pytest plugin entrypoint (bubblegum.pytest_plugin) that can emit HTML/JSON reports and manage artifacts during test sessions.

Common commands

# HTML report
pytest --bubblegum-config bubblegum.yaml \
  --bubblegum-report artifacts/bubblegum-report.html

# JSON report
pytest --bubblegum-config bubblegum.yaml \
  --bubblegum-report-json artifacts/bubblegum-report.json

# HTML + JSON + explicit artifacts directory
pytest --bubblegum-config bubblegum.yaml \
  --bubblegum-artifacts artifacts \
  --bubblegum-report artifacts/bubblegum-report.html \
  --bubblegum-report-json artifacts/bubblegum-report.json

# Run benchmark validation at session end
pytest --bubblegum-benchmark

GitHub Actions example

- name: Run tests with Bubblegum reporting
  run: |
    pytest \
      --bubblegum-config bubblegum.yaml \
      --bubblegum-artifacts artifacts \
      --bubblegum-report artifacts/bubblegum-report.html \
      --bubblegum-report-json artifacts/bubblegum-report.json

- name: Upload Bubblegum artifacts
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: bubblegum-artifacts
    path: artifacts/

Project status note (Object Intelligence planning)

• Phase 19A audit concluded: architectural strength is strong alpha and empirical capability remains unrated until benchmark implementation exists.
• Phase 19B is docs/design-only and defines Object Intelligence Benchmark vs Regression Suite separation for v0.0.6-alpha planning.
• See: docs/phase-19b-object-intelligence-benchmark.md.

Quickstart examples

Runnable templates are available in:

• examples/playwright_quickstart.py
• examples/appium_quickstart.py
• examples/hybrid_web_mobile_example.py
• examples/README.md

These are intentionally minimal and avoid credentials/secrets.

The Playwright quickstart uses page.set_content(...) with deterministic local HTML by default, so first-run smoke does not depend on outbound network access.

The Appium quickstart is a real-environment template (Appium server + emulator/device + app/capabilities) and is intentionally not self-contained.

For quickstart troubleshooting (dependency/proxy issues, Playwright browser setup, Appium server/device requirements), see examples/README.md.

For Appium-specific onboarding (real-environment prerequisites, capability checklist, and common startup failures), jump to the Appium setup and troubleshooting sections in examples/README.md.

For hybrid web + mobile usage patterns (selector-first + natural-language fallback), see examples/hybrid_web_mobile_example.py.

For callable vision provider lifecycle setup/teardown with required privacy gates, see examples/vision_callable_provider_example.py and docs in docs/phase-11l-callable-vision-enablements.md.

Adoption/docs MVP links:

• Adoption guide: docs/adoption.md
• Web natural-language quickstart: examples/web_nl_quickstart.py
• OCR callable hydration pattern: examples/ocr_callable_hydration_example.py
• Report artifacts example: examples/report_artifacts_example.py
• Pytest plugin usage: docs/pytest-plugin.md
• CI usage snippet: docs/ci.md
• Real-environment smoke harness commands (opt-in, skip-by-default): tests/real_env/README.md

Distribution posture note:

• Published to PyPI as bubblegum-ai (latest: 0.0.6a0 / v0.0.6-alpha) via Trusted Publishing, alongside GitHub pre-release tags.
• An @bubblegum-ai/node npm client is planned so TS/JS Playwright projects can drive the same engine — see docs/distribution-npm-and-pypi.md.

Optional OpenAI vision backend note:

• OpenAIVisionProvider is available as an optional vision backend under bubblegum.core.vision.backends.
• It requires either an injected compatible client or user-installed OpenAI SDK (not required by base install).
• Provider-based screenshot-to-vision invocation is runtime cost-gated and requires max_cost_level="high" in execution options plus all vision/privacy gates.
• Manually supplied vision_candidates remain allowed at any cost level and are not overwritten by SDK provider wiring.
• Phase 11T hardening: provider model must be non-empty and timeout must be a positive number.
• timeout is applied when using optional lazy SDK client creation (create_client=True); injected client remains preferred for deterministic tests.
• It is only used when you explicitly call configure_vision_provider(...) and all vision/privacy gates are enabled.
• No raw screenshot bytes are persisted to traces/metadata by Bubblegum vision plumbing.
• OpenAIVisionProvider now exposes sanitized failure diagnostics (last_diagnostic / get_last_diagnostic()), with stable code/stage metadata and no raw screenshot bytes/base64/request payload/API key leakage.
• Manual real-provider usage example: examples/openai_vision_provider_manual_example.py (optional/manual; user installs openai; the OpenAI SDK reads OPENAI_API_KEY from environment).

Pytest reporter fixture integration

If you want Bubblegum HTML/JSON reports to include step-level SDK results, add the bubblegum_reporter fixture and append each StepResult:

import pytest
from bubblegum import act

@pytest.mark.asyncio
async def test_login_flow(page, bubblegum_reporter):
    result = await act("Click Login", channel="web", page=page)
    bubblegum_reporter.add(result)
    assert result.status in {"passed", "recovered"}

Run with report outputs:

pytest --bubblegum-config bubblegum.yaml \
  --bubblegum-report artifacts/bubblegum-report.html \
  --bubblegum-report-json artifacts/bubblegum-report.json

Phase 11L callable vision guide

For a complete callable vision enablement guide (gates, output shape, troubleshooting, and future provider lifecycle/API audit), see:

• docs/phase-11l-callable-vision-enablements.md

Phase 11N public provider lifecycle API

Bubblegum now exposes a minimal public runtime lifecycle API for optional vision wiring:

from bubblegum import configure_vision_provider, clear_vision_provider
from bubblegum.core.vision.backends.callable import CallableVisionProvider

configure_vision_provider(CallableVisionProvider(my_vision_callable))
# ... run steps ...
clear_vision_provider()  # idempotent reset

Guardrails remain unchanged:

• Registration does not request screenshots or invoke providers by itself.
• Provider invocation remains default-off and requires all gates: enable_vision, send_screenshots, process_screenshots_for_vision, and registered provider.
• Manual vision_candidates still take precedence over provider output.
• Provider exceptions remain fail-safe (no hard grounding crash from provider errors).

Optional vision callable backend (no bundled vision model dependency)

Bubblegum does not bundle a real vision model dependency by default. You can provide a runtime callable backend via CallableVisionProvider that receives screenshot bytes, instruction text, and optional context, and returns raw vision candidates (list[VisionCandidate] or list[dict]).

• Screenshot processing remains opt-in and privacy-gated.
• enable_vision: true alone does not process screenshots.
• send_screenshots: true grants screenshot capture/sharing permission.
• process_screenshots_for_vision: true is also required before screenshot bytes should be processed into vision candidates.
• SDK runtime now supports optional internal screenshot-to-vision candidate wiring, but it is strictly disabled by default and runs only when all gates are true: enable_vision, send_screenshots, process_screenshots_for_vision, and a configured runtime vision provider/callable.
• VisionModelResolver behavior is unchanged: it consumes injected intent.context["vision_candidates"] only.
• If vision_candidates are already user-supplied in context, SDK wiring does not overwrite them.
• No raw screenshot bytes are persisted into resolver metadata/traces by the SDK wiring path.
• Vision refs remain synthetic (vision://target/<index>) and are not adapter-executed.
• No real bundled vision model dependency is added; runtime callable/provider remains integrator-supplied.

Optional OCR callable backend (no bundled OCR dependency)

Bubblegum does not bundle a real OCR engine dependency by default. You can provide a runtime callable backend that receives screenshot bytes and returns raw OCR blocks. The callable contract is:

• Input: image_bytes: bytes (raw screenshot bytes)
• Output: list[OCRBlock] or list[dict]
• Canonical block shape after normalization:

{
    "text": str,
    "bbox": [x1, y1, x2, y2],
    "confidence": float,
}

from bubblegum.core.ocr.backends import CallableOCREngine
from bubblegum.core.ocr.engine import build_ocr_blocks_from_screenshot


def my_ocr_callable(image_bytes: bytes):
    # Integrator-owned OCR logic here
    # Return list[OCRBlock] or list[dict(text,bbox,confidence)]
    return [{"text": "Continue", "bbox": [10, 20, 100, 60], "confidence": 0.93}]


engine = CallableOCREngine(my_ocr_callable)
ocr_blocks = build_ocr_blocks_from_screenshot(
    screenshot=b"...png bytes...",
    enabled=True,
    process_screenshots_for_ocr=True,
    engine=engine,
)

ocr_blocks output is normalized into canonical context shape and is safe-by-default:

• Screenshot OCR processing runs only when process_screenshots_for_ocr: true is explicitly enabled.
• Bubblegum does not bundle a real OCR engine dependency yet.
• OCRResolver consumes normalized context["ocr_blocks"] and emits synthetic ocr://block/<index> refs.
• These ocr:// refs are not adapter-executed yet (resolution/ranking signal only in current architecture).

Public API

Four primitives. These are the only methods test authors need.

# Natural language — direct mode (web)
await act("Click Login",          channel="web")
await verify("Dashboard visible", channel="web")
await extract("Get user email",   channel="web")

# Natural language — mobile
await act("Tap Continue",   channel="mobile", platform="android")
await verify("OTP screen",  channel="mobile", platform="ios")

# Fallback recovery — plug into an existing Playwright test
await recover(
    page=page,
    failed_selector="#submit-btn",
    intent="Click Submit"
)

Mobile recover(): recover() supports mobile by passing channel="mobile" and an Appium driver.

StepResult — what every call returns

class StepResult(BaseModel):
    status:      Literal["passed", "failed", "recovered", "skipped"]
    action:      str
    target:      ResolvedTarget | None
    confidence:  float
    validation:  ValidationResult | None
    artifacts:   list[ArtifactRef]
    duration_ms: int
    error:       ErrorInfo | None
    traces:      list[ResolverTrace]   # one entry per resolver that ran

"recovered" status flags that the original selector failed but Bubblegum fixed it — important for CI reporting so your team knows which steps need their locators updated.

How It Works

Execution Pipeline (10 steps)

Confidence Scoring

Thresholds

All thresholds are configurable in bubblegum.yaml.

Resolver Chain

Built-in Resolvers

Tiered Execution

Tier 1 — deterministic (priority 0–39):
  ExplicitSelectorResolver, MemoryCacheResolver,
  AccessibilityTreeResolver, AppiumHierarchyResolver,
  ExactTextResolver
  → If confidence ≥ 0.85: stop here.

Tier 2 — fuzzy / semantic (priority 40–49):
  FuzzyTextResolver
  → If confidence ≥ 0.70: stop here.

Tier 3 — AI fallback (priority 50+):
  LLMGroundingResolver, OCRResolver, VisionModelResolver
  → Only reached when Tiers 1+2 cannot meet review_threshold.
  → Blocked entirely if max_cost_level: low in config.

Memory Self-Healing

On repeat CI runs, MemoryCacheResolver replays successful mappings without any AI call. Before replaying, it verifies:

• Screen signature still matches (fingerprint comparison within tolerance)
• Element still exists in current DOM/hierarchy
• Element text, role, and position are within drift threshold
• Last success was within TTL (default: 7 days)
• Failure count is below max_failures (default: 3)

If any check fails, it steps aside gracefully and lets a downstream resolver win.

Step Artifact (per resolved step)

{
  "step":             "Click Login",
  "channel":          "web",
  "resolvers_tried":  ["explicit_selector", "memory_cache", "accessibility_tree"],
  "candidates": [
    { "resolver": "accessibility_tree", "ref": "button[name='Login']",
      "confidence": 0.94, "matched_text": "Login" },
    { "resolver": "exact_text", "ref": "text=Login", "confidence": 0.86 }
  ],
  "selected":         { "resolver": "accessibility_tree", "confidence": 0.94 },
  "ambiguous":        false,
  "tier_stopped_at":  1,
  "duration_ms":      43,
  "screenshot":       "artifacts/step_001.png"
}

Configuration

Full bubblegum.yaml reference:

grounding:
  accept_threshold:    0.85
  review_threshold:    0.70
  ambiguous_gap:       0.05
  reject_threshold:    0.50
  max_cost_level:      medium   # low | medium | high
  enable_vision:       false    # must be true to use VisionModelResolver
  enable_ocr:          true
  memory_ttl_days:     7
  memory_max_failures: 3

ai:
  enabled:   true
  provider:  anthropic          # anthropic | openai | gemini | local
  model:     <your-model-name>  # e.g. claude-sonnet-latest, gpt-4o, gemini-pro

privacy:
  redact_pii:         true
  send_screenshots:   false
  log_provider_calls: true

debug:
  log_raw_payloads:    false   # never enable in CI or production
  log_resolver_traces: true    # safe — logs resolver names + confidence only

Fully offline / deterministic mode:

ai:
  enabled: false   # disables all AI resolvers, no model provider required

Error Handling

All errors extend BubblegumError and carry: step instruction, resolver name, candidates found, and a screenshot artifact reference.

Safe failure over wrong click. Bubblegum prefers raising AmbiguousTargetError over executing on a low-confidence match. Clicking the wrong element silently is worse than a clear, informative failure.

Privacy & Security

Because Bubblegum may send DOM text and hierarchy data to external LLM providers:

• Only the filtered accessibility tree or hierarchy XML is sent — not the full raw DOM
• Screenshots are sent to the vision model only when VisionModelResolver is explicitly enabled
• Phase 11D adds an injected-candidate MVP in VisionModelResolver: it consumes intent.context["vision_candidates"] (normalized), emits synthetic vision://target/<index> refs, and remains deterministic with no bundled real vision model dependency.
• Vision refs remain synthetic/non-executable in this phase; no adapter execution path for vision:// refs is wired yet.
• Password fields (input[type=password]) are never included in any snapshot
• Fields labelled secret, token, api_key, or similar are auto-redacted
• Email and phone number redaction is configurable via privacy.redact_pii: true
• Every AI provider call is logged with safe metadata only — provider name, resolver name, token count, redaction status, response latency — not the raw prompt or screenshot content
• Raw payload logging is available only in debug.log_raw_payloads: true mode — never enable in CI

Local / offline mode: Set ai.enabled: false to disable all AI resolvers. Local model providers (e.g. Ollama) are supported via provider: local. The memory layer uses local SQLite only — no remote sync unless explicitly configured.

Roadmap

• Latest release: v0.0.6-alpha — first PyPI publish + JSON-RPC engine bridge (0.1.0).
• In progress: the @bubblegum-ai/node npm client (0.2.0) — see docs/distribution-npm-and-pypi.md.
• Current focus: web/mobile automation and smarter, faster, cheaper AI grounding.
• Paused (maintenance mode): the code-generator features — bubblegum record (capture → NL steps) and bubblegum convert (manual scenarios → scaffolds). No new feature work is planned; both remain available and supported for existing users with unchanged CLI/config/output. Bug fixes only.
• Earlier planning context: docs/phase-17a-roadmap-reset-v0.0.5-alpha.md.

Planned v0.0.5-alpha scope themes:

1. release/packaging confidence,
2. deterministic quality gates,
3. docs/operator clarity,
4. no SDK/API/schema breaking changes.

Benchmark Strategy

Current benchmark status (Phase 8 baseline)

Current deterministic benchmark expectations:

• static validation: 12/12 passed
• execute validation: total 12, executed 12, skipped 0, passed 12, failed 0

Static validation checks fixture schema, snapshot existence, and static expected winner/confidence ranges. Execute validation runs deterministic benchmark execution and evaluates execute_* expectations, which may intentionally differ from static expectations.

execute_allow_review=true is benchmark-only review-pass handling and does not change SDK/engine runtime semantics. Deterministic execution excludes Tier 3 AI/OCR/Vision resolvers. Memory benchmark runs with ephemeral DB setup/pre-seeding and does not use .bubblegum/memory.db.

Benchmark commands:

python scripts/run_benchmarks.py
python scripts/run_benchmarks.py --execute

A golden dataset is created in Phase 0. Without it, Bubblegum cannot be proven better than plain Playwright.

Dataset Composition (100 scenarios)

• 50 standard web UI steps — buttons, forms, links, dropdowns, modals
• 20 broken-selector cases — element exists but selector is stale/wrong
• 10 changed-label cases — button text changed slightly (Login → Sign In)
• 10 duplicate-label cases — multiple matching elements on page
• 10 dynamic/modal/overlay cases — element appears after an interaction

Metrics Tracked

• Deterministic success rate (Tier 1–2 resolver wins)
• AI recovery success rate (Tier 3 resolver wins)
• False positive rate (wrong element selected)
• Ambiguous target rate (AmbiguousTargetError frequency)
• Average step latency per resolver (ms)
• Model calls per scenario (cost proxy)
• Estimated cost per test run (USD via token counting)
• Resolver win distribution by action type

Contributing Custom Resolvers

Zero core changes required. Ship a resolver as a standalone class:

from bubblegum.core.grounding.resolver import Resolver, StepIntent, ResolvedTarget

class MaterialUIResolver(Resolver):
    name       = "material_ui"
    priority   = 25          # runs between AccessibilityTreeResolver and ExactTextResolver
    channels   = ["web"]
    cost_level = "low"
    tier       = 1

    def required_context(self): return ["a11y_snapshot"]
    def supports(self, intent): return intent.action_type in ["click", "type", "select"]

    def resolve(self, intent: StepIntent) -> list[ResolvedTarget]:
        # MUI-specific matching logic
        ...

Register it in one line (e.g. in conftest.py):

bubblegum.registry.register(MaterialUIResolver())

Community resolvers can be published as separate packages:

pip install bubblegum-salesforce-resolver
pip install bubblegum-flutter-resolver

Architecture

Repository Structure

bubblegum/
├── core/
│   ├── grounding/
│   │   ├── engine.py           # GroundingEngine orchestrator
│   │   ├── registry.py         # ResolverRegistry
│   │   ├── resolver.py         # Resolver abstract base
│   │   ├── ranker.py           # CandidateRanker + confidence formula
│   │   ├── confidence.py       # Threshold logic
│   │   ├── errors.py           # Full error taxonomy
│   │   └── resolvers/          # One file per resolver
│   ├── schemas.py              # All 13 Pydantic schemas
│   ├── parser/                 # NL instruction → StepIntent
│   ├── planner/                # StepIntent → ActionPlan
│   ├── validation/             # ValidationEngine
│   ├── recovery/               # RecoveryEngine
│   ├── memory/                 # MemoryLayer (SQLite)
│   └── models/                 # LLM provider abstraction
├── adapters/
│   ├── web/playwright/         # PlaywrightAdapter
│   └── mobile/appium/          # AppiumAdapter
├── reporting/                  # HTML/JSON report, pytest plugin
└── tests/
    ├── unit/                   # Each resolver tested in isolation
    ├── integration/
    ├── e2e/
    └── benchmarks/             # Golden dataset runner

Schema Catalogue (13 Pydantic Models)

Release/package validation

These checks are intended for maintainers preparing an MVP release candidate (not required for normal package users).

CI-required release checks (current workflow):

# offline-safe package validation (default)
python scripts/validate_package.py

# strict maintainer/release-mode validation
python scripts/validate_package.py --strict

# build artifacts
python -m build

Optional maintainer check (local, if twine is installed):

python -m twine check dist/*

Publishing is automated via PyPI Trusted Publishing in .github/workflows/publish.yml (manual dispatch → TestPyPI; v* tag → PyPI). See docs/publishing.md.

Project Status

_{Bubblegum · Architecture Candidate v0.9 — Final Approved · April 2026}