truffles 0.1.0

Tools for data extraction from the web with AI

truffles 🍫

A python framework that extends playwright with language model tools (page.tools) for web automation and data extraction.

Note: Currently only supported with async playwright.

CI/CD
Meta

An open-source framework that extends Playwright with AI capabilities for web automation and data extraction. truffles implements global caching of AI interactions to improve efficiency across the community.

page = truffles.wrap(browser)

# go to your favorite website
page.goto("https://example.com/products")

# auto list detection
items = await page.tools.get_main_list()

# json structure from locator
products = await items.tools.to_structure(
    {
        "title": str,
        "price": float
    }
)

Tools

The .tools attribute is a namespace for all added tools, cleanly separating them from the page object.

Implemented Tools

The repo contains a few tools that implement common patterns for web automation, that are powered by LLMs and have been proven to work well in the wild. Currently, the following tools are implemented or in the works:

• Automatic List Detection
• Autonomous Pagination (under development)
• Intelligent & Safe Data Extraction (under development)

Context Manager

The ContextManager is internally used by tools to store and retrieve reusable context in automations. This greatly reduces the amount of LLM calls required. In the future, a publicly shared context will be available as an optional feature to further reduce costs. The ContextManager operates mostly in the background and is automatically initialized when truffles.astart is called, but it can also be initialized and accessed manually.

from truffles.context import ContextManager

ContextManager.initialize()

Tool Extensibility

The framework is designed to be easily extensible. You can implement your own tools by inheriting from BaseTool and implementing the execute method.

from truffles.tools import BaseTool

class MyTool(BaseTool):

    def execute(self, *args, **kwargs):

        # your code here

        pass

Technical Details

• Asynchronous architecture
• Built to seamlessly extend existing playwright projects
• Out-of-the-box support for langchain supported models

A more extensive setup example

from playwright.async_api import async_playwright
import truffles

p = await async_playwright().start()
browser = await p.chromium.launch()

# initializes the default model and context manager
await truffles.astart()
page = truffles.wrap(browser)

# your code here

Project Status

Current development focuses on:

• Improving current tools
• Extending the functionality
• Optimizing performance of ContextManager