Geonode Scraper Tools Core

Shared runtime, schemas, and operation registry for Geonode Scraper tool integrations.

Most users should install one of the framework packages instead:

• geonode-scraper-langchain
• geonode-scraper-crewai

Install the core package directly only if you are building your own wrapper layer on top of the shared service.

Installation

pip install geonode-scraper-tools-core

Public API

• ScraperToolSettings
• ScraperToolService
• OperationSpec
• OPERATIONS
• get_operations()

Configuration

from geonode_scraper_tools_core import ScraperToolSettings, ScraperToolService

settings = ScraperToolSettings(
    host="https://api.example.com",
    api_key="your-api-key",
)

service = ScraperToolService(settings)

Exposed Operations

The shared service normalizes SDK responses into JSON-friendly dictionaries and exposes the following 21 operations:

Extraction

• extract — extract content from a single URL (sync or async)
• get_job_result — fetch the current state or result of an async extraction job
• wait_for_job — poll an async extraction job until it reaches a terminal state
• list_jobs — list previously submitted extraction jobs with optional filters

Batch

• create_batch — submit a list of URLs for asynchronous batch extraction
• get_batch_status — poll the current status and partial results of a batch job
• wait_for_batch — poll a batch job until it reaches a terminal state
• list_batch_jobs — list previously submitted batch jobs with optional filters

Crawl

• create_crawl — start a crawl job from a seed URL
• get_crawl_status — poll the current status and results of a crawl job
• wait_for_crawl — poll a crawl job until it reaches a terminal state
• list_crawl_jobs — list previously submitted crawl jobs with optional filters

Map

• map_urls — discover all URLs under a base URL via sitemap and HTML link extraction
• list_map_jobs — list previously submitted map jobs with optional filters
• get_map_job — fetch the status and discovered URLs for a single map job

Search

• search — run a search query and return ranked results
• list_search_jobs — list previously submitted search jobs with optional filters
• get_search_job — fetch the status and results for a single search job

Statistics, Usage & Health

• get_statistics — retrieve aggregated extraction statistics
• get_concurrency_usage — check the caller's live work-concurrency slot usage against their plan limit
• health_check — check the scraper service health and version

Selecting a Subset of Operations

Pass an operations list to get_operations() or to any framework wrapper to expose only the operations you need.

from geonode_scraper_tools_core import get_operations

ops = get_operations(["extract", "map_urls", "create_batch", "wait_for_batch"])