truffles 0.2.0

Tools for data extraction from the web with AI

truffles 🍫

An open-source framework that extends playwright with automatic element detection, validation and structuring. Further utilities such as global caching of selectors reduce latency and cost.

Note: Currently only supported with async playwright.

Installation

$ pip install truffles

After installation you may need to run playwright install. A more extensive introduction to the package can be found here.

Quick Start

A common workflow that is fully automated by truffles is data extraction from list elements:

import truffles
from pydantic import BaseModel

await truffles.start()

### define a schema ###
class Product(BaseModel):
    title: str = Field(description = "The name of the product")
    price: float = Field(description = "The main listed price")


### Define & Execute Task ###
task = truffles.tasks.ListingTask(
    page="https://example.com/products",
    schema=Product
)

results = await task.run()

Tools

The page.tools and locator.tools currently extend playwright with automatic list detection and structured element extraction. This makes all truffles code

• Natively integrated with playwright

• Reusable across websites

• Much more robust to page content and structure changes

An Example Listing Task

page = truffles.wrap(browser)
page.goto("https://example.com/products")


item_list = await page.tools.get_main_list() # automatically finds main list on page


task_list = [
    loc.tools.to_structure(Product) # structures locator content according to Product
    for loc in locators
]
await asyncio.gather(task_list)

Tool Overview

The repo contains tools that implement common patterns for web automation. They are commonly powered by LLMs and have been extensively tested. These are:

• Automatic List Detection (see get_main_list)

• Intelligent & Robust Data Extraction (see to_structure)

• Get Element by Prompt (under development)

• Page to Markdown (under development)

• Support for Image Data (under development)

Extensibility

Truffles is designed to be easily extensible. You can implement your own tools by inheriting from BaseTool and implementing the execute method.

from truffles.tools import BaseTool

class ValidateElementAlignment(BaseTool):

    def execute(self, alignment_rule, prompt = None):

        # your code here
        pass

The Store Manager

The StoreManager is internally used by tools to cache and retrieve reusable context in automations. This greatly reduces the amount of LLM calls required.

It operates mostly in the shadows and is automatically initialized when truffles.start is called, but it can also be initialized and accessed manually.

from truffles.context import StoreManager

StoreManager.initialize()

In the future, a publicly accessible context will be available to further reduce costs.

Project Status

Current development focuses on:

• Improving current tools
• Extending the functionality
• Optimizing performance of StoreManager