Sonde

Probe any HTTP API for its rate limits, burst ceiling, and full-scrape time. Provider-pluggable, safe by default.

Install

Requires Python 3.12+.

pip install sonde

From source:

git clone https://github.com/Jartan-LLC/sonde.git
cd sonde
pip install -e .

For Docker, see Docker below.

Quick Start

Probe the Roblox asset-owners endpoint:

export ROBLOX_COOKIE="your_roblosecurity_cookie"
sonde asset-owners --asset-id 20573078 --total-items 1470000

Probe GitHub stargazers:

export GITHUB_TOKEN="ghp_..."
sonde github-stargazers --owner Jartan-LLC --repo sonde

Without GITHUB_TOKEN, sonde probes anonymously, at GitHub's lower rate limit.

Results are written to sonde_report.json by default:

sonde asset-owners --asset-id 20573078 --output my_report.json

How It Works

Sonde runs five phases against the target endpoint, then combines the measurements into a safe rate estimate.

How the estimate is produced

The estimate phase uses a priority ladder to determine the safe rate:

1. Authoritative headers -- If the API returned x-ratelimit-limit and a window, use those directly (e.g. 100 requests per 60s).
2. Swept floor -- If the sweep found a fastest sustainable interval, use that.
3. Token-bucket inference -- If burst results show a clean burst size and a measured recovery window, infer the bucket rate.
4. Sequential fallback -- Use the observed sequential throughput before the first 429.
5. No-throttle fallback -- If nothing ever throttled, no ceiling was found, so fall back to a conservative fraction of the measured sequential throughput.

Every rung applies the safety margin (default 80%, configurable with --margin) -- the recommended pace is ~25% slower than the measured ceiling. Rung 5 has no measured ceiling, so it applies an extra 0.5 factor on top (~40% of observed throughput at the default margin).

Endpoints

asset-owners

Roblox inventory.roblox.com/v2/assets/{id}/owners -- paginated list of owners of a collectible asset.

Auth: Set ROBLOX_COOKIE (legacy web-session) and/or ROBLOX_BEARER (Open Cloud) environment variables.

github-stargazers

GitHub api.github.com/repos/{owner}/{repo}/stargazers -- users who starred a repository.

Auth: Set GITHUB_TOKEN environment variable. Without it, you get the anonymous rate limit (60 requests/hour).

Adding an Endpoint

Write the endpoint in your own package. Once the package is installed alongside sonde, the endpoint is a sonde subcommand.

1. Subclass Endpoint, set a unique name (the subcommand) and a one-line help, and implement build_request(cursor) and parse_page(response).

2. Override make_provider() to return the API's Provider: the generic one handles standard 200/429 and IETF headers, and GitHubProvider and RobloxProvider are built in. A provider that sends credentials in headers lists them in credentials(), so logs redact them; query-parameter credentials go in its auth_params(), not in an endpoint's request parameters.

3. Optionally implement total_items() for scrape-time estimates, add_arguments() / from_args() for CLI options, and extra_headers() for endpoint-specific headers.

4. If the endpoint is paginated, call add_pagination_args(parser, page_max=...) in add_arguments() and pagination_from_args(args, page_max=...) in from_args(), with the API's largest page size, so it gets the shared --page-size / --total-items flags (clamped to that size).

5. Declare the class in your package's pyproject.toml:

   [project.entry-points."sonde.endpoints"]
   my-endpoint = "my_package.endpoints:MyEndpoint"
   

Import what you need from sonde: its exports are the plugin API. While sonde is 0.x, a minor release can change them, and the changelog says how.

Minimal example:

from typing import Any, override

from sonde import Endpoint, PageResult, RequestSpec


class MyEndpoint(Endpoint):
    name = "my-endpoint"
    help = "one-line description for --help"

    @override
    def build_request(self, cursor: Any) -> RequestSpec:
        return RequestSpec(url="https://api.example.com/items", params={"page": cursor or 1})

    @override
    def parse_page(self, response: Any) -> PageResult:
        data = response.json()
        return PageResult(count=len(data["items"]), next_cursor=data.get("next_page"))

A built-in endpoint lives in src/sonde/endpoints/ instead, decorated with @register and imported in that package's __init__.py.

CLI Reference

Common options shared by all endpoints:

-v and -q are mutually exclusive. Logs always go to stderr; the report goes to --output.

Exit codes: 0 success, 2 precondition failure (bad arguments, an endpoint plugin that won't load, unwritable --output, or the endpoint returned no usable response), 1 unexpected crash, 130 interrupted.

Piping and machine-readable output

Use --output - to write the JSON report to stdout instead of a file. Combine with -q to suppress INFO-level log noise on stderr:

sonde asset-owners --asset-id 20573078 --output - -q | jq .estimate

Use --log-format json for structured log lines on stderr (keys: timestamp, level, logger, message, plus exc on error lines), useful for log aggregators or CI pipelines:

sonde asset-owners --asset-id 20573078 --log-format json 2>sonde.log

Docker

Build:

docker build -t sonde .

Run (mount current directory so the report lands on the host):

docker run --rm -v "$(pwd):/data" -e ROBLOX_COOKIE sonde \
    asset-owners --asset-id 20573078 --total-items 1470000

docker run --rm -v "$(pwd):/data" -e GITHUB_TOKEN sonde \
    github-stargazers --owner Jartan-LLC --repo sonde

The container writes sonde_report.json to /data (the mounted volume).

Development

Setup and the checks to run before a pull request are in CONTRIBUTING.md.

License

MIT