Skip to main content

Vision utilities for web interaction agents

Project description

Tarsier Monkey

🙈 Vision utilities for web interaction agents 🙈

Python Version

🔗 Main site   •   🐦 Twitter   •   📢 Discord

Tarsier

If you've tried using GPT-4(V) to automate web interactions, you've probably run into questions like:

  • How do you map LLM responses back into web elements?
  • How can you mark up a page for an LLM better understand its action space?
  • How do you feed a "screenshot" to a text-only LLM?

At Reworkd, we found ourselves reusing the same utility libraries to solve these problems across multiple projects. Because of this we're now open-sourcing this simple utility library for multimodal web agents... Tarsier! The video below demonstrates Tarsier usage by feeding a page snapshot into a langchain agent and letting it take actions.

https://github.com/reworkd/tarsier/assets/50181239/af12beda-89b5-4add-b888-d780b353304b

How does it work?

Tarsier works by visually "tagging" interactable elements on a page via brackets + an id such as [1]. In doing this, we provide a mapping between elements and ids for GPT-4(V) to take actions upon. We define interactable elements as buttons, links, or input fields that are visible on the page.

Can provide a textual representation of the page. This means that Tarsier enables deeper interaction for even non multi-modal LLMs. This is important to note given performance issues with existing vision language models. Tarsier also provides OCR utils to convert a page screenshot into a whitespace-structured string that an LLM without vision can understand.

Installation

pip install tarsier

Usage

Visit our cookbook for agent examples using Tarsier:

Otherwise, basic Tarsier usage might look like the following:

import asyncio

from playwright.async_api import async_playwright
from tarsier import Tarsier, GoogleVisionOCRService
import json

def load_google_cloud_credentials(json_file_path):
    with open(json_file_path) as f:
        credentials = json.load(f)
    return credentials

async def main():
    # To create the service account key, follow the instructions on this SO answer https://stackoverflow.com/a/46290808/1780891
    google_cloud_credentials = load_google_cloud_credentials('./google_service_acc_key.json')

    ocr_service = GoogleVisionOCRService(google_cloud_credentials)
    tarsier = Tarsier(ocr_service)

    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False)
        page = await browser.new_page()
        await page.goto("https://news.ycombinator.com")

        page_text, tag_to_xpath = await tarsier.page_to_text(page)

        print(tag_to_xpath)  # Mapping of tags to x_paths
        print(page_text)  # My Text representation of the page


if __name__ == '__main__':
    asyncio.run(main())

Keep in mind that Tarsier tags different types of elements differently to help your LLM identify what actions are performable on each element. Specifically:

  • [#ID]: text-insertable fields (e.g. textarea, input with textual type)
  • [@ID]: hyperlinks (<a> tags)
  • [$ID]: other interactable elements (e.g. button, select)
  • [ID]: plain text (if you pass tag_text_elements=True)

Local Development

Setup

We have provided a handy setup script to get you up and running with Tarsier development.

./script/setup.sh

If you modify any TypeScript files used by Tarsier, you'll need to execute the following command. This compiles the TypeScript into JavaScript, which can then be utilized in the Python package.

npm run build

Testing

We use pytest for testing. To run the tests, simply run:

poetry run pytest .

Linting

Prior to submitting a potential PR, please run the following to format your code:

./script/format.sh

Supported OCR Services

Roadmap

  • Add documentation and examples
  • Clean up interfaces and add unit tests
  • Launch
  • Improve OCR text performance
  • Add options to customize tagging styling
  • Add support for other browsers drivers as necessary
  • Add support for other OCR services as necessary

Citations

bibtex
@misc{reworkd2023tarsier,
  title        = {Tarsier},
  author       = {Rohan Pandey and Adam Watkins and Asim Shrestha and Srijan Subedi},
  year         = {2023},
  howpublished = {GitHub},
  url          = {https://github.com/reworkd/tarsier}
}

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tarsier-0.5.90.tar.gz (14.5 kB view details)

Uploaded Source

Built Distribution

tarsier-0.5.90-py3-none-any.whl (15.7 kB view details)

Uploaded Python 3

File details

Details for the file tarsier-0.5.90.tar.gz.

File metadata

  • Download URL: tarsier-0.5.90.tar.gz
  • Upload date:
  • Size: 14.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.7.1 CPython/3.10.12 Linux/6.2.0-1019-azure

File hashes

Hashes for tarsier-0.5.90.tar.gz
Algorithm Hash digest
SHA256 45e10aa9f0a9dd21bf37e3e1ce94c1de4c03e2e14e4e31d1fa4d154916d470d9
MD5 1a4527e24c421c67d3c481a6a25fb151
BLAKE2b-256 47640dbc141feeaba09da3aff2f20f7125ba32de7bad2a1679afadad85b88b47

See more details on using hashes here.

File details

Details for the file tarsier-0.5.90-py3-none-any.whl.

File metadata

  • Download URL: tarsier-0.5.90-py3-none-any.whl
  • Upload date:
  • Size: 15.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.7.1 CPython/3.10.12 Linux/6.2.0-1019-azure

File hashes

Hashes for tarsier-0.5.90-py3-none-any.whl
Algorithm Hash digest
SHA256 b5b063d0526486c2b1b7c50c8b8da4d95958e1140c227146578193fb70705e28
MD5 96d8419ed1959c9f975bd8fd200761ec
BLAKE2b-256 a3118ab7a7c28ef554add468b6056b0ee1ad6102ea3218064da94e24b1676073

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page