Geonode Python SDK: Web Scraping & Data Extraction

Build Python agents and data pipelines with Geonode’s typed clients. Extract pages, crawl sites, discover URLs, search the web and run batches. Use returned content in your own research and RAG workflows, with webhook subscriptions and usage statistics available from your code.

Try Geonode for free

Integrations page · Geonode API docs · Python SDK setup guide · View package on PyPI · View source on GitHub · Complete guide

Get started · Example: prepare website content for retrieval · Setup questions

Workflow ideas

• Prepare documents for retrieval. Extract Markdown from selected URLs, then pass it to your own chunking, embedding and storage pipeline.
• Refresh a website dataset. Crawl with depth and page limits, inspect results, and save content with its source URL.

Get started

Requires Python 3.10 or newer and a Geonode API key. API requests use your Geonode account's plan and balance.

pip install geonode-scraper-sdk

Set GEONODE_SCRAPER_API_KEY in your server environment, then run:

import os
from geonode_scraper_sdk import (
    ApiClient, Configuration, ExtractionApi, ExtractRequest, OutputFormat,
)

configuration = Configuration(
    api_key={"ApiKeyAuth": os.environ["GEONODE_SCRAPER_API_KEY"]}
)
with ApiClient(configuration) as client:
    result = ExtractionApi(client).extract_v1_extract_post(
        ExtractRequest(url="https://example.com", formats=[OutputFormat.MARKDOWN])
    )
    print(result.data.markdown)

The SDK sends your key in X-Api-Key to https://scraper.geonode.io. Keep the key out of source control and browser code.

What you can build

See the complete guide for background extraction, batch and crawl examples, proxy settings, webhooks, error handling and the method reference. Generated method names follow the API contract. The SDK is an API client; it does not bundle a local browser or MCP server.

Understand the extraction result

The Python quick start requests Markdown and reads data.markdown from a completed extraction. Request HTML when you need data.html. Keep metadata.url alongside the content so later processing retains its source. tokens_charged reports the request's usage.

A background request returns a job ID and status instead of immediately returning finished content. Keep that ID, retrieve the job and inspect its terminal status before reading the result. A timeout after submission is not a reason to submit the same job again.

Example: prepare website content for retrieval

1. Start with a small list of source URLs, or map a site and select the relevant pages.
2. Extract Markdown from one URL and inspect its content and source metadata before processing the list.
3. Use a batch for a known URL list, or a crawl with explicit depth and page limits when you need linked pages.
4. Pass completed content to your own chunking and embedding pipeline, then store the chunks with their source URLs and retrieval time.
5. Handle failed pages separately so partial results do not silently become a complete dataset.

Expected result: Page content and source metadata are ready for your application. Embeddings, vector storage, product-field parsing and change detection are separate steps you supply.

Setup questions

Missing content? Confirm the requested format and whether the job has completed. Enable JavaScript rendering when the target depends on a browser, and inspect the returned status and error details.

Where does the API key belong? Keep it in your server environment and pass it through the SDK configuration. A browser application should call your own server instead of exposing the key.

Does installing the SDK run a scraper locally? It installs an API client. Extraction runs through the hosted Geonode service and uses your account's allowance.

Need MCP? Use the separate hosted MCP connection for a compatible assistant; this SDK does not start an MCP server.

These recipes describe source-supported workflows. They have not been run as authenticated end-to-end examples in this review.

Development

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev]' build twine
bash scripts/ruff.sh check
bash scripts/test.sh
python -m build
python -m twine check dist/*

Read CONTRIBUTING.md for the workflow, code generation and releases.

Support and license

Use GitHub issues for SDK bugs and Geonode support for account or API questions. Report vulnerabilities privately using SECURITY.md. Released changes are listed in CHANGELOG.md.

MIT license · Copyright 2026 Geonode Team.

Try Geonode for free