swarmauri_tool_webscraping 0.11.0.dev1

Swarmauri web-content extraction tool for scraping HTML pages with CSS selectors.

Swarmauri Tool Web Scraping

swarmauri_tool_webscraping is a Swarmauri web-content extraction tool that fetches a page with httpx, parses HTML with BeautifulSoup, and extracts text using a CSS selector. It is useful for headline capture, policy checks, lightweight data extraction, and agent workflows that need webpage content on demand.

Why Use Swarmauri Tool Web Scraping

• Extract targeted text from webpages using CSS selectors.
• Add lightweight HTML scraping to Swarmauri agents and automation flows.
• Pull site copy, headlines, notices, or metadata for downstream analysis.
• Return structured extraction or error results without custom scraping glue.

FAQ

What inputs does the tool expect?
A url string and a CSS selector string.

What does the tool return?
Either {"extracted_text": ...} or {"error": ...}.

What happens when no elements match?
The tool returns an empty extracted_text string.

Does it render JavaScript-driven pages?
No. It only fetches raw HTTP content and parses returned HTML.

Features

• Swarmauri ToolBase implementation registered as WebScrapingTool.
• Uses standard CSS selectors to target page elements.
• Returns joined text content across all selector matches.
• Handles request and parsing failures with structured error output.
• Supports Python 3.10, 3.11, 3.12, 3.13, and 3.14.

Installation

uv add swarmauri_tool_webscraping

pip install swarmauri_tool_webscraping

Usage

from swarmauri_tool_webscraping import WebScrapingTool

tool = WebScrapingTool()
result = tool(url="https://example.com", selector="h1")

print(result)

Examples

Extract a page headline

from swarmauri_tool_webscraping import WebScrapingTool

tool = WebScrapingTool()
result = tool("https://example.com", "h1")

print(result.get("extracted_text"))

Inspect a status banner

from swarmauri_tool_webscraping import WebScrapingTool

tool = WebScrapingTool()
result = tool("https://status.example.com", ".banner")

if "error" not in result:
    print(result["extracted_text"])

Register the tool in a Swarmauri collection

from swarmauri_standard.tools.ToolCollection import ToolCollection
from swarmauri_tool_webscraping import WebScrapingTool

tools = ToolCollection(tools=[WebScrapingTool()])
print(tools)

Related Packages

• swarmauri_tool_downloadpdf
• swarmauri_tool_searchword
• swarmauri_tool_zapierhook

Swarmauri Foundations

• swarmauri
• swarmauri_core
• swarmauri_base
• swarmauri_standard

More Documentation

• HTTPX documentation
• Beautiful Soup documentation
• MDN CSS selectors reference

Best Practices

• Respect site terms, rate limits, and robots rules before scraping.
• Use stable selectors and expect sites to change their markup over time.
• Prefer dedicated APIs when a provider offers one.
• Extend the tool if you need headers, retries, or authenticated requests.

License

This project is licensed under the Apache-2.0 License.