Skip to main content

Vision utilities for web interaction agents

Project description

Tarsier Monkey

🙈 Vision utilities for web interaction agents 🙈

Python Version

🔗 Main site   •   🐦 Twitter   •   📢 Discord

Tarsier

If you've tried using an LLM to automate web interactions, you've probably run into questions like:

  • How should you feed the webpage to an LLM? (e.g. HTML, Accessibility Tree, Screenshot)
  • How do you map LLM responses back to web elements?
  • How can you inform a text-only LLM about the page's visual structure?

At Reworkd, we iterated on all these problems across tens of thousands of real web tasks to build a powerful perception system for web agents... Tarsier! In the video below, we use Tarsier to provide webpage perception for a minimalistic GPT-4 LangChain web agent.

https://github.com/reworkd/tarsier/assets/50181239/af12beda-89b5-4add-b888-d780b353304b

How does it work?

Tarsier visually tags interactable elements on a page via brackets + an ID e.g. [23]. In doing this, we provide a mapping between elements and IDs for an LLM to take actions upon (e.g. CLICK [23]). We define interactable elements as buttons, links, or input fields that are visible on the page; Tarsier can also tag all textual elements if you pass tag_text_elements=True.

Furthermore, we've developed an OCR algorithm to convert a page screenshot into a whitespace-structured string (almost like ASCII art) that an LLM even without vision can understand. Since current vision-language models still lack fine-grained representations needed for web interaction tasks, this is critical. On our internal benchmarks, unimodal GPT-4 + Tarsier-Text beats GPT-4V + Tarsier-Screenshot by 10-20%!

Tagged Screenshot Tagged Text Representation
tagged tagged

Installation

pip install tarsier

Usage

Visit our cookbook for agent examples using Tarsier:

Otherwise, basic Tarsier usage might look like the following:

import asyncio

from playwright.async_api import async_playwright
from tarsier import Tarsier, GoogleVisionOCRService
import json

def load_google_cloud_credentials(json_file_path):
    with open(json_file_path) as f:
        credentials = json.load(f)
    return credentials

async def main():
    # To create the service account key, follow the instructions on this SO answer https://stackoverflow.com/a/46290808/1780891
    google_cloud_credentials = load_google_cloud_credentials('./google_service_acc_key.json')

    ocr_service = GoogleVisionOCRService(google_cloud_credentials)
    tarsier = Tarsier(ocr_service)

    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False)
        page = await browser.new_page()
        await page.goto("https://news.ycombinator.com")

        page_text, tag_to_xpath = await tarsier.page_to_text(page)

        print(tag_to_xpath)  # Mapping of tags to x_paths
        print(page_text)  # My Text representation of the page


if __name__ == '__main__':
    asyncio.run(main())

Keep in mind that Tarsier tags different types of elements differently to help your LLM identify what actions are performable on each element. Specifically:

  • [#ID]: text-insertable fields (e.g. textarea, input with textual type)
  • [@ID]: hyperlinks (<a> tags)
  • [$ID]: other interactable elements (e.g. button, select)
  • [ID]: plain text (if you pass tag_text_elements=True)

Local Development

Setup

We have provided a handy setup script to get you up and running with Tarsier development.

./script/setup.sh

If you modify any TypeScript files used by Tarsier, you'll need to execute the following command. This compiles the TypeScript into JavaScript, which can then be utilized in the Python package.

npm run build

Testing

We use pytest for testing. To run the tests, simply run:

poetry run pytest .

Linting

Prior to submitting a potential PR, please run the following to format your code:

./script/format.sh

Supported OCR Services

Roadmap

  • Add documentation and examples
  • Clean up interfaces and add unit tests
  • Launch
  • Improve OCR text performance
  • Add options to customize tagging styling
  • Add support for other browsers drivers as necessary

Citations

bibtex
@misc{reworkd2023tarsier,
  title        = {Tarsier},
  author       = {Rohan Pandey and Adam Watkins and Asim Shrestha and Srijan Subedi},
  year         = {2023},
  howpublished = {GitHub},
  url          = {https://github.com/reworkd/tarsier}
}

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tarsier-0.5.95.tar.gz (15.4 kB view details)

Uploaded Source

Built Distribution

tarsier-0.5.95-py3-none-any.whl (16.5 kB view details)

Uploaded Python 3

File details

Details for the file tarsier-0.5.95.tar.gz.

File metadata

  • Download URL: tarsier-0.5.95.tar.gz
  • Upload date:
  • Size: 15.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.10.12 Linux/6.5.0-1021-azure

File hashes

Hashes for tarsier-0.5.95.tar.gz
Algorithm Hash digest
SHA256 43f973f18879229c5411f4593c1e859c59aff4ef376616b5d36c4b92f131cb13
MD5 107c872e141f50ecdec66352c60cf7d7
BLAKE2b-256 402cf5c08720678e3667c71de56f510aadba9ace555c5e0e792dea6e75cf59d0

See more details on using hashes here.

File details

Details for the file tarsier-0.5.95-py3-none-any.whl.

File metadata

  • Download URL: tarsier-0.5.95-py3-none-any.whl
  • Upload date:
  • Size: 16.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.3 CPython/3.10.12 Linux/6.5.0-1021-azure

File hashes

Hashes for tarsier-0.5.95-py3-none-any.whl
Algorithm Hash digest
SHA256 c802de2f251de75bf78b442f815cced2c4f49a5b9c4b58f286a9004f7d6ae4e7
MD5 aa85e0ea0beeb5cf758f0100354770c5
BLAKE2b-256 01841d5c2f55a51f6ff5b019ac2b46cf434235b064e240a5623871db7d00c38d

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page