Vision utilities for web interaction agents
Project description
🙈 Vision utilities for web interaction agents 🙈
🔗 Main site • 🐦 Twitter • 📢 Discord
Tarsier
If you've tried using GPT-4(V) to automate web interactions, you've probably run into questions like:
- How do you map LLM responses back into web elements?
- How can you mark up a page for an LLM better understand its action space?
- How do you feed a "screenshot" to a text-only LLM?
At Reworkd, we found ourselves reusing the same utility libraries to solve these problems across multiple projects. Because of this we're now open-sourcing this simple utility library for multimodal web agents... Tarsier!
How does it work?
Tarsier works by visually "tagging" interactable elements on a page via brackets + an id such as [1]
.
In doing this, we provide a mapping between elements and ids for GPT-4(V) to take actions upon.
We define interactable elements as buttons, links, or input fields that are visible on the page.
Can provide a textual representation of the page. This means that Tarsier enables deeper interaction for even non multi-modal LLMs. This is important to note given performance issues with existing vision language models. Tarsier also provides OCR utils to convert a page screenshot into a whitespace-structured string that an LLM without vision can understand.
Installation
pip install tarsier
Usage
An agent using Tarsier might look like this:
from playwright.async_api import async_playwright
from tarsier import Tarsier, GoogleVisionOCRService
async def main():
google_cloud_credentials = {...}
ocr_service = GoogleVisionOCRService(google_cloud_credentials)
tarsier = Tarsier(ocr_service)
async with async_playwright() as p:
browser = await p.chromium.launch(headless=False)
page = await browser.new_page()
await page.goto("https://news.ycombinator.com")
driver = tarsier.create_driver(page)
page_text, tag_to_xpath = await tarsier.page_to_text(driver)
print(page_text) # My Text representation of the page
print(tag_to_xpath) # Mapping of tags to x_paths
Visit our cookbook for additional examples:
- A LangChain web agent
- A LlamaIndex web agent
Roadmap
-
Add documentation and examples
-
Clean up interfaces and add unit tests
-
Launch
-
Improve OCR text performance
-
Add options to customize tagging
-
Add support for other browsers drivers as necessary
-
Add support for other OCR services as necessary
Citations
bibtex
@misc{reworkd2023tarsier,
title = {Tarsier},
author = {Rohan Pandey and Adam Watkins and Asim Shrestha and Srijan Subedi},
year = {2023},
howpublished = {GitHub},
url = {https://github.com/reworkd/bananalyzer}
}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.