Geonode CLI: Web Scraping, Crawling & Search

Collect web data from your terminal for scripts, agents and data pipelines. Extract pages, crawl sites, discover URLs, search the web and run batches with the Geonode CLI. JSON output and exit codes make the results easy to use in your own automation.

Try Geonode for free

Integrations page · Geonode API docs · CLI setup guide · View package on PyPI · View source on GitHub · Complete guide

Quick start · Choose a command · Example: save a page for a scheduled pipeline · Setup questions

Workflow ideas

• Schedule a content refresh. Run extraction commands from your scheduler, inspect exit codes, and save JSON results.
• Inspect a site from the terminal. Map URLs, select pages and extract Markdown without writing an API client.

Quick start

Requires Python 3.10 or newer and a Geonode API key from the dashboard.

pipx install geonode-scraper-cli
export GEONODE_SCRAPER_API_KEY="your-api-key"
gscraper extract https://example.com --format markdown

Use pip install geonode-scraper-cli if you already manage your own virtual environment. On Windows PowerShell, set $env:GEONODE_SCRAPER_API_KEY = "your-api-key".

For machine-readable output:

gscraper extract https://example.com --format markdown --json

A successful response has ok: true and the API result under result; a failed request has ok: false and an error object. API requests require an active account and use your plan's allowance.

Choose a command

Use gscraper --help or gscraper COMMAND --help for options. See the complete guide for background jobs, profiles, output files, shell completion and exit codes.

Configuration and errors

Set credentials through the environment or a local config file; command-line keys can remain in shell history. gscraper config show masks keys. Flags override environment variables, which override the selected profile.

A missing/invalid key or blocked target exits with code 4; invalid request/extraction failure uses 6, a network failure uses 7, and a wait timeout uses 8. See all exit codes when handling failures in scripts.

Example: save a page for a scheduled pipeline

After configuring GEONODE_SCRAPER_API_KEY, run one extraction and save its structured response. This Bash example writes to a local file; it does not create a scheduler or a storage service.

if gscraper extract https://example.com --format markdown --json > page-result.json; then
  echo "Inspect ok and result.data.markdown in page-result.json"
else
  echo "Extraction failed; inspect page-result.json" >&2
fi

Expected result: Successful synchronous JSON output has ok: true and content under result.data.markdown. Inspect failure output before importing it into a dataset.

Setup questions

Command not found? Check that the environment where you installed the package is active and that its Scripts or bin directory is on PATH. pipx manages a separate command environment.

Wrong account or endpoint? Inspect gscraper config show, which masks keys. Flags override environment variables, which override the selected configuration profile.

Need reliable scheduling? Let your scheduler inspect the command's exit code, preserve failed output and apply a bounded retry policy. A submitted background job still needs a completion check.

Does the CLI run an MCP server? No. It calls the Geonode API; the hosted MCP endpoint is a separate assistant connection.

These recipes describe source-supported workflows. They have not been run as authenticated end-to-end examples in this review.

Development and support

See CONTRIBUTING.md for local checks and releases.

For API or account questions, contact Geonode. For security issues, follow SECURITY.md. Licensed under MIT.

Try Geonode for free