Skip to main content

Undetected Browser Tool

Publish Python 🐍 distribution 📦 to PyPI and TestPyPI

Introduction

The Undetected Browser Tool is an AI Agent tool to simplify the process of accessing individual web pages that are protected by bot detection systems.

e.g. Cloudflare and similar services.

This tool is designed to facilitate access to publicly available data on a single-page basis for AI Agents. This is not a hacking tool, or a paywall bypass tool, nor guaranteed in all situations.

It is not intended for web scraping or crawling purposes

Various configurations in WAF and security services may still block access, and the tool can still be detected through methods like the Chrome Developer Protocol (CDP) in JavaScript.

It works by creating a headless browser instance and navigating to a page, which inherently makes it slower and unsuitable for high-volume or automated data extraction.

The main aim is to make it easier for AI agents to access web pages that might otherwise be challenging due to bot protection.

It is built as a LangChain tool using Selenium and the undetected-chromedriver project.

Installation

pip install -U undetected-browser-tool

Usage

We use this in langchain agents or in CrewAI Agents

from undetected_browser_tool import UndetectedBrowserTool

# for a headless browser
browser = UndetectedBrowserTool(headless=True) 

# or useful for debugging
browser = UndetectedBrowserTool(headless=False) 


# or using a proxy, change http to https or socks5 to suit your proxy settings
opts = {"proxy-server" : "http://xxx.xxx.xxx:port"}
browser = UndetectedBrowserTool(headless=False, additional_opts=opts) 

# fetch a page
page = browser.run("https://nowsecure.nl/")

Example with CrewAI

Untested code, written as an example for usage with CrewAI Be aware that langchain and Crew are in constant development, so these interfaces may change.

from crewai import Agent, Task, Crew
from undetected_browser_tool import UndetectedBrowserTool
from langchain_community.tools import DuckDuckGoSearchResults

browser = UndetectedBrowserTool(headless=True)
search_engine = DuckDuckGoSearchResults()

researcher = Agent(
            role="Researcher",
            goal="A document reviewing the top 5 solutions in {topic}",
            backstory="You are a research analyst tasked with reviewing software for an IT firm to help them make buying decisions",
            tools=[search_engine, browser],
)

report_writer = Agent(
            role="Report writer",
            goal="A executive review of the top 5 solutions in {topic}"
            backstory="You are a report writer for an IT firm, you excel at writing summaries and detailed reports for executives in an IT firm", 
            
)

research_task = Task(
            expected_output="A list of pros and cons, features, customer reviews and pricing of the top 5 solutions in {topic}",
            description="By using a search engine and Capterra, G2, Gartner write a review of the top 5 solutions in {topic}, include the source link for each item you find."
            tools=[search_engine, browser],
            agent=researcher,
)

report_task = Task(
            expected_output="An executive style report, with a summary, detailed information and a recommendation for selecting a solution in {topic}"
            description="Based upon the research provided, write an executive summary, detailed report, and a recommendation on the software selection. The report should have a table of features, pros / cons, and pricing options."
            output_file="report.md",
            context={research_task}
)

crew = Crew(agents=[researcher, report_writer], tasks=[research_task, report_task])

crew.kickoff(inputs={"topic" : "BI Reporting Platforms"})

Example of using with smolagents

Smolagents can use langchain tools via

from undetected_browser_tool import UndetectedBrowserTool
from langchain_community.agent_toolkits.load_tools import load_tools

from smolagents import CodeAgent, LiteLLMModel, Tool, ToolCallingAgent

model = LiteLLMModel(model_id=model_id)
tool_list = ["ddg-search", "wikipedia"] # "wolfram-alpha"
AUTHORIZED_IMPORTS = [
                      "requests",
                      "pandas",
                      "numpy",
                      "json",
                      "bs4",
                      "datetime",
                      "csv",
                      "pathlib",
                      "os",
                      "io"
                  ]

# Create some tools for the agent
tools = []
for tool in tool_list: 
  tools.append(
    Tool.from_langchain(load_tools([tool], allow_dangerous_tools=True)[0])
    )
    

# Add undetected browser        
tools.append(
    Tool.from_langchain(UndetectedBrowserTool()))

agent = CodeAgent(tools=tools, 
          model=model,
          additional_authorized_imports=AUTHORIZED_IMPORTS,
          max_steps=10,
          add_base_tools=True)

result = agent.run("Find me some good tacos near Reston, VA")

Tips for being undetected

This is not a 100%, it's hard to be 100% but it's a good start.

  • Don't go through a data center, those IPs are easy to track and block.
  • Use proxies, and I recommend residential proxies. Tor networks are easily blocked.
  • Switch proxies frequently
  • Don't crawl the website, use a search service to try and pinpoint the page you need.

Every time a confirm you are a human page pops up, about 50% of the users just drop out, so companies who are dependent on traffic try to avoid that. Meaning as long as you're not doing something silly you're usually going to be ok.

Ethics

The use of automation tools to interact with websites is a nuanced topic that involves balancing access to information with respect for publishers' rights. Publishers have the right to protect their content and manage how it is accessed. They often allow search engines to index their data so it can be discovered by the general public, while also setting guidelines to regulate how their information is used.

If your intent is to occasionally access content to obtain information and properly reference the source, this tool can help streamline your workflow by automating repetitive tasks, thus saving time and effort.

However, if your goal is to scrape large amounts of data from a website, this project is not intended for that purpose. Engaging in large-scale scraping can violate the terms of service of many websites, infringe on intellectual property rights, and potentially cause harm to the website's infrastructure.

Always ensure that your use of automation tools is ethical and complies with the website's terms of service. Respect the rights of content owners and use automation responsibly.

Metadata

Release files for undetected-browser-tool 1.1.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for undetected-browser-tool 1.1.3
File Size Uploaded Provenance Actions
undetected_browser_tool-1.1.3.tar.gz 7.0 kB With provenance Details for undetected_browser_tool-1.1.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for undetected-browser-tool 1.1.3
File Interpreter ABI Platform Actions
undetected_browser_tool-1.1.3-py3-none-any.whl Python 3 none any Details for undetected_browser_tool-1.1.3-py3-none-any.whl

Total release size: 14.7 kB

Release files / undetected_browser_tool-1.1.3.tar.gz

Download URL undetected_browser_tool-1.1.3.tar.gz
Size 7.0 kB
Tags Source
SHA-256 checksum
How to use checksums
d1cfa6ce12a977665c7316238b590ffa16be544d6016e786ee16d7082fed9002
BLAKE2b-256 checksum
How to use checksums
cd4ffb63ad4ecf5ae34992327ee68f4dfa0c7ff05fb896f6520819788f96fb0a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.12.9

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Feb 21, 2025.

Transparency log

Release files / undetected_browser_tool-1.1.3-py3-none-any.whl

Download URL undetected_browser_tool-1.1.3-py3-none-any.whl
Size 7.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
78c7a351d57a05def02971abc7f5f7a706231a98e39217891fe27767425e6049
BLAKE2b-256 checksum
How to use checksums
a629b51e0b4a261066924f61f09420574f8d521184ab7d8cd90220ded44c02d2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.12.9

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Feb 21, 2025.

Transparency log

Security for undetected-browser-tool 1.1.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for undetected_browser_tool-1.1.3.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Feb 21, 2025.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page