lite-locator-mcp

Playwright MCP with a semantic element finder in front. The proxy runs Playwright MCP as a child process and exposes all of its tools unchanged, plus browser_find_element(step). That tool takes the accessibility snapshot inside the proxy, ranks the elements against the test step with small local models, and returns only a shortlist with refs. The LLM acts on a ref (browser_click target=e990) and calls browser_snapshot only when nothing in the shortlist fits.

Measured on the hard held-out test set (2,875 steps, 38 sites never seen in training), the correct element was in a shortlist of about 150–500 tokens 82–90% of the time. The full page averaged 12,229 tokens. Playwright MCP is started with --snapshot-mode none, so clicks and typing no longer return a full snapshot each time.

Install

Needs Python 3.10+ and Node 18+ (for Playwright MCP itself).

pip install lite-locator-mcp        # or: uv tool install lite-locator-mcp
npx playwright install chromium                           # once, if no browser is installed

Use in an MCP client

.mcp.json (Claude Code, Cursor, etc.):

{"mcpServers": {"playwright": {"command": "lite-locator-mcp", "args": ["--", "--headless", "--isolated"]}}}

Arguments after -- are passed to Playwright MCP. --lam sets the cut policy (0.002, 0.005, 0.01 or 0.02; a higher value shows fewer candidates; the default is 0.01).

Environment variables:

• PLAYWRIGHT_MCP overrides the child command (default npx -y @playwright/mcp@0.0.83).
• ELEM_RANK_LITE points to a different model bundle.

Build the wheel (from the repo)

.venv/bin/python -m elem_rank.export_onnx --out elem_rank/models-lite   # ONNX int8 models, 57 MB
.venv/bin/python -m build      # sdist + wheel in dist/

Credits and licences

• Code: MIT (see LICENSE).
• Retriever fine-tuned from BAAI/bge-small-en-v1.5 (MIT).
• Reranker fine-tuned from cross-encoder/ms-marco-MiniLM-L6-v2 (Apache-2.0).
• Training data includes Mind2Web (Deng et al., 2023, CC BY 4.0) and pages from public websites.