Geonode for CrewAI: Web Scraping Tools for AI Agents

Give your CrewAI researchers tools to find and collect source material from the web. Search the web, discover URLs, extract pages, crawl sites and run batches. Feed the returned content into your crew’s analysis and reporting tasks, with tools to retrieve or wait for background jobs.

Try Geonode for free

Integrations page · Geonode API docs · CrewAI setup guide · View package on PyPI · View source on GitHub · Complete guide

Quick start · Available operations · Example: a research workflow in CrewAI · Setup questions

Workflow ideas

• Support a research crew. Give a researcher tools to find and extract sources, then pass those results to a separate analysis task.
• Refresh knowledge inputs. Collect selected website pages as Markdown before your crew analyzes or summarizes them.

Quick start

Requires Python 3.10 or newer and a Geonode API key from the dashboard. The direct-tool example works without a language model.

pip install geonode-scraper-crewai
export GEONODE_SCRAPER_API_KEY="your-api-key"

import os

from geonode_scraper_crewai import build_crewai_tools
from geonode_scraper_tools_core import ScraperToolSettings

settings = ScraperToolSettings(
    host="https://scraper.geonode.io",
    api_key=os.environ["GEONODE_SCRAPER_API_KEY"],
)
tools = build_crewai_tools(settings=settings, operations=["extract"])
extract = next(tool for tool in tools if tool.name == "scraper_extract_content")
outcome = extract.run(url="https://example.com", formats=["markdown"])

if outcome["ok"]:
    print(outcome["result"]["data"]["markdown"])
else:
    print(outcome["error"]["status"], outcome["error"]["message"])

API calls use your Geonode plan. Successful tools return ok, operation, attempts and result; API failures return a structured error with status, message and retryability. Validation or framework errors may still raise before an API request.

Use with an agent

Install your model provider's integration and set its credentials separately. Set CREWAI_MODEL to a model identifier supported by your installed CrewAI version. Some model providers require a CrewAI extra; see the complete guide.

Continue from the configured tools above:

from crewai import Agent, Crew, Task

researcher = Agent(
    role="Web researcher",
    goal="Read web pages and report what they say",
    backstory="You check sources before answering.",
    tools=tools,
    llm=os.environ["CREWAI_MODEL"],
)
task = Task(
    description="Read https://example.com and summarize the page.",
    expected_output="One concise paragraph based on the page.",
    agent=researcher,
)
print(Crew(agents=[researcher], tasks=[task]).kickoff())

The model chooses tool calls. Restrict the available operations to the tasks your agent needs:

tools = build_crewai_tools(
    settings=settings,
    operations=["extract", "map_urls", "search"],
)

Available operations

The complete guide lists each tool ID, settings, polling and response handling. The shared geonode-scraper-tools-core runtime is installed automatically. This package integrates directly with CrewAI; it does not start an MCP server.

Understand the tool result

The toolkit wraps successful API results with ok, operation, attempts and result. For synchronous Markdown extraction, check ok before reading result.data.markdown. API failures return an error with status, code, message and retryable; argument-validation or framework errors can raise before that envelope is returned.

The quick start calls a tool directly, without a model. Once that succeeds, give the selected tools to your agent and configure its model provider separately. The Geonode key and model-provider key are different credentials.

Example: a research workflow in CrewAI

1. Choose only the extract, map_urls and search operations for a focused research task.
2. Ask the agent to discover relevant URLs, select a small set, and extract those pages as Markdown.
3. Require it to inspect tool errors and retain the source URL for every page it uses.
4. Pass the extracted content to the model for analysis, then save the answer with its sources in your own application.
5. Use explicit page limits and a separate completion check if the task needs a crawl or batch.

Expected result: The tools return web content and job results. The model performs the reasoning and summarization; these toolkits do not supply a vector database, a model subscription or a ready-made research agent.

Setup questions

Why is the agent not calling a tool? First run the direct-tool example, then check that your agent receives the selected tools and that its model supports tool calling. Installing the package alone does not configure an agent.

Why did a long job time out? Request timeouts and wait-tool polling timeouts are separate. Preserve the job ID and retrieve its state before retrying a submission.

Do I need agent tools core separately? The framework package installs the shared runtime as a dependency. Use it directly only when building your own adapter.

Is this MCP? These are native framework tools. Use the separate Geonode MCP connection for an MCP-compatible assistant.

These recipes describe source-supported workflows. They have not been run as authenticated end-to-end examples in this review.

Development and support

See CONTRIBUTING.md for local checks and releases.

For API or account questions, contact Geonode. For security issues, follow SECURITY.md. Licensed under MIT.

Try Geonode for free