Dendrite
Dendrite is an SDK that makes it easy to interact with and scrape websites using prompts.
# Extracting data from a page is as simple as doing page.scrape(prompt)
startup_urls = await page.scrape("Extract a list of valid urls for each listed startup's YC page")
Join our discord to give feedback and report bugs! Invite link
Installation:
pip install dendrite-python-sdk && playwright install
Quick start:
Here is a very simple example of how to use Dendrite.
Install dendrite and asyncio like so: pip install asyncio dendrite-python-sdk && playwright install and create a file called main.py. Use the code below and run it with python main.py from the terminal.
main.py
from dendrite_python_sdk import DendriteBrowser
import asyncio
async def main():
# Launch browser locally
dendrite_browser = DendriteBrowser(openai_api_key=...) # Use your OpenAI key here
# Go to google.com
page = await dendrite_browser.goto("https://google.com")
# Get the reject cookies button
close_cookies = await page.get_interactable_element("reject cookies popup button")
# Click the button, returns IncorrectOutcomeException if the popup wasn't closed
await close_cookies.click(expected_outcome="That the cookies popup closed.")
# Get the search bar
search_bar = await page.get_interactable_element("Return the search bar")
# Enter "hello world" into it.
await search_bar.fill(
"hello world", expected_outcome="The words 'hello world' to have been entered"
)
# Use the Playwright page to press enter.
await page.get_playwright_page().keyboard.press("Enter")
await asyncio.sleep(2)
await dendrite_browser.close()
asyncio.run(main())
More Advanced Example:
With Dendrite we can do more than just finding and using elements.
We can scrape data from pages and ensure that the returned data followed a certain structure by prompting, using a JSON schema or passing in a pydantic model.
Below is an example of how he can, see more examples in dendrite_python_sdk/examples.
import os
import asyncio
from dendrite_python_sdk import DendriteBrowser
from dotenv import load_dotenv, find_dotenv
import pandas as pd
load_dotenv(find_dotenv())
async def main():
dendrite_browser = DendriteBrowser(
openai_api_key=os.environ.get("OPENAI_API_KEY", ""),
dendrite_api_key=os.environ.get("DENDRITE_API_KEY", "") # Generate a dendrite API at https://dendrite.se to get better rate limits and latency.
)
page = await dendrite_browser.goto(
"https://ycombinator.com/companies", scroll_through_entire_page=False
)
# Get elements with prompts instead of using brittle selectors
search_bar = await page.get_interactable_element(
"The search bar used to find startups"
)
await search_bar.fill("AI agent")
# Scraping data is as easy as writing a good prompt
await page.scroll_through_entire_page()
startup_urls = await page.scrape(
"Extract a list of valid urls for each listed startup's YC page"
)
# Let's go to every AI agent startup's page and fetch information about the founders
startup_info = []
for url in startup_urls:
page = await dendrite_browser.goto(url)
startup_details = await page.scrape(
"Extract a dict like this: {name: str, slogan: str, founders_social_media_urls: a string where each url is separated by a linebreak}"
)
print("startup_details: ", startup_details)
startup_info.append(startup_details)
df = pd.DataFrame(startup_info)
df.to_excel("ai_agent_founders.xlsx", index=False)
asyncio.run(main())
Class Documentation
DendriteBrowser
DendriteBrowser is an abstraction on top of playwright's browser. You can do everything you would normally do with Playwright but with a few new handy AI functions. Access
Methods
__init__(self, openai_api_key: str, id=None, playwright_options: Any = {"headless": False})
Initializes the DendriteBrowser class.
Params:
openai_api_key(str): Your OpenAI API Key.id(Optional): The unique ID for the browser session.playwright_options(dict): Options to configure Playwright browser.
get_active_page(self) -> DendritePage
Returns the active page.
goto(self, url: str, scroll_through_entire_page: Optional[bool] = True) -> DendritePage
Navigates to the specified URL.
Params:
url(str): URL of the webpage.scroll_through_entire_page(Optional[bool]): Whether to scroll through the entire page.
google_search(self, query: str, filter_results_prompt: Optional[str] = None, load_all_results: Optional[bool] = True) -> List[SearchResult]
Performs a Google search and returns search results.
Params:
query(str): The search query.filter_results_prompt(Optional[str]): Prompt to filter results.load_all_results(Optional[bool]): Whether to load all results.
new_page(self) -> DendritePage
Opens a new page in the browser.
close(self)
Closes the browser context.
DendritePage
Abstraction layer on top of Playwright's Page class. Use get_playwright_locator to get the playwright page.
Methods
get_playwright_page(self) -> Page
Returns the Playwright page instance.
scrape(self, prompt: str, expected_return_data: Optional[str] = None, return_data_json_schema: Optional[Any] = None, pydantic_return_model: Optional[Type[BaseModel]] = None) -> Any
Scrapes data from the page based on a prompt.
Params:
prompt(str): The prompt to gather data.expected_return_data(Optional[str]): Expected return data.return_data_json_schema(Optional[Any]): JSON schema for return data.pydantic_return_model(Optional[Type[BaseModel]]): Pydantic model for return data.
scroll_through_entire_page(self) -> None
Scrolls through the entire page.
get_interactable_element(self, prompt: str) -> DendriteLocator
Returns an interactable element based on a prompt.
DendriteLocator
Abstraction layer on top of Playwright's Locator class. Use get_playwright_locator to get the playwright locator.
Methods
get_playwright_locator(self) -> Locator
Returns the Playwright locator.
wait_for_page_changes(self, old_url: str, timeout: float = 2)
Waits for changes in the page. Params:
old_url(str): The URL before the interaction.timeout(float): The time to wait for changes.
click(self, expected_outcome="", *args, **kwargs) -> InteractionResponse
Clicks the element and handles the interaction.
Params:
expected_outcome(str): Expected outcome after clicking.
fill(self, value: str, expected_outcome="", *args, **kwargs)
Fills the element with the provided value.
Params:
value(str): The value to fill.expected_outcome(str): Expected outcome after filling.
Metadata
Release files for dendrite-python-sdk 1.0.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| dendrite_python_sdk-1.0.2.tar.gz | 17.9 kB | Without provenance | Details for dendrite_python_sdk-1.0.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| dendrite_python_sdk-1.0.2-py3-none-any.whl | Python 3 | none | any | Details for dendrite_python_sdk-1.0.2-py3-none-any.whl |
Total release size: 43.4 kB
Release files / dendrite_python_sdk-1.0.2.tar.gz
| Download URL | dendrite_python_sdk-1.0.2.tar.gz |
|---|---|
| Size | 17.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c9da0037e26cd6b334a9c566a371f2f60ed8a0448d742522147721231ad35cf0
|
|
BLAKE2b-256 checksum How to use checksums |
c3290dbbbc01ecbfe9ada42bb12d1803903c2e86cf2195ad7ab1acc0f2f8dd52
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.8.3 CPython/3.12.2 Darwin/23.5.0
|
Release files / dendrite_python_sdk-1.0.2-py3-none-any.whl
| Download URL | dendrite_python_sdk-1.0.2-py3-none-any.whl |
|---|---|
| Size | 25.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f47ff6e17ccff8f592b507bfbc7ea3e0f846eb931663f7799e677bc7f58d0e3f
|
|
BLAKE2b-256 checksum How to use checksums |
0867e73a1ac4e1f15390cf5d3a0d8ccfcfa38691a4303e50a95b6e4956568cfd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.8.3 CPython/3.12.2 Darwin/23.5.0
|
Security for dendrite-python-sdk 1.0.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.