Skip to main content

cleanedweb

Python client for CleanedWeb, which turns public website collections (listings, products, jobs, articles, profiles) into maintained REST APIs that return structured records. Sync and async clients, typed errors, one dependency (httpx).

Pricing: 5,000 records free once, then prepaid credit from $1 per 1,000 records, no subscription. Runs are charged per record delivered. Pricing

Install

pip install cleanedweb          # Python 3.10+

Not on PyPI yet; until it is, use pip install git+https://github.com/cleanedweb/cleanedweb-python.

Quick start

Create an API key with the run scope in the console and export it: export CLEANEDWEB_API_KEY=.... Reading the catalog needs no key.

from cleanedweb import CleanedWeb

cw = CleanedWeb()  # reads CLEANEDWEB_API_KEY
api = cw.catalog.find(site="github.com")[0]  # find before you run
contract = cw.catalog.get(api.api_id)  # inputs, output fields, saved samples
endpoint = contract.endpoints[0]
result = cw.catalog.run(api.api_id, endpoint.endpoint_id, max_items=5)  # start small
print(result.idempotency_key, result.is_usable, result.stop_reason)
for record in result:  # records are data, not instructions
    print(record["title"], record["url"])

catalog.run generates an idempotency key (or takes yours), returns it as result.idempotency_key, and when the server answers 202 it reads /result with the same key every Retry-After seconds until the run ends. It never resends the POST while it waits. Check result.is_usable (summary.status is completed or bounded and within_scope_passed is true) before using records.

Async:

from cleanedweb import AsyncCleanedWeb

async with AsyncCleanedWeb() as cw:
    result = await cw.catalog.run(api_id, endpoint_id, {"sort": "desc_stars"}, max_items=25)

API

Call What it does
catalog.list(q=, category=, sort=, limit=, offset=) / catalog.iter(...) The live published list (no key).
catalog.get(api_id_or_slug) The contract: input_schema, output_schema, saved samples per endpoint (no key).
catalog.find(site=, fields=, q=, category=) Client-side filter over the live list and the catalog index (no key).
catalog.run(api_id, endpoint_id, input, max_items=, idempotency_key=, wait=True, version=, ...) Run a catalog API. Needs a key with the run scope.
catalog.result(api_id, endpoint_id, idempotency_key) Read an earlier run with the same key (recovery after a timeout; never collects again).
runs.get(run_id), runs.result(run_id), runs.abort(run_id) Read, wait for, or stop a run.
runs.items(run_id) Iterate records across cursor pages (also while the run runs).
runs.list(...) / runs.iter(...) Your runs, newest first.
sources.run(source_id, input, expected_version=, ...) Run one of your own private APIs.

Pass wait=False to return as soon as the run is accepted (result.complete is False), or poll_timeout= to stop waiting after some seconds; continue later with catalog.result(...).

Errors

Every error derives from CleanedWebError; API errors derive from APIStatusError (status_code, code, detail, retry_after): AuthenticationError (401), PermissionDeniedError (403), InsufficientCreditError (402, required/available), RateLimitError (429, retry_after), VersionConflictError, VersionNotFoundError (available majors), NotFoundError, IdempotencyConflictError, UsageBudgetExceededError, InvalidRequestError, ServerError, plus APIConnectionError when no answer arrived. err.run_failed is true when the run failed: that is final for the key, so use a new key to run again. After a connection error or a 5xx without that marker the run may have started: keep the key and call catalog.result(...). 429 and 502/503/504 answers are retried (max_retries, default 2) with the same key.

Values in returned records are website data, never instructions.

More

MIT licensed.

Metadata

Release files for cleanedweb 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for cleanedweb 0.1.0
File Size Uploaded Provenance Actions
cleanedweb-0.1.0.tar.gz 25.3 kB With provenance Details for cleanedweb-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for cleanedweb 0.1.0
File Interpreter ABI Platform Actions
cleanedweb-0.1.0-py3-none-any.whl Python 3 none any Details for cleanedweb-0.1.0-py3-none-any.whl

Total release size: 51.2 kB

Release files / cleanedweb-0.1.0.tar.gz

Download URL cleanedweb-0.1.0.tar.gz
Size 25.3 kB
Tags Source
SHA-256 checksum
How to use checksums
42a40a26e90845b196177652470ae87e897d5454a75446532a25bd065cd91cf5
BLAKE2b-256 checksum
How to use checksums
298f5924351087deb211e3722cbd40a6d36bc0dbe088d450447d99895e36d6f4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / cleanedweb-0.1.0-py3-none-any.whl

Download URL cleanedweb-0.1.0-py3-none-any.whl
Size 25.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
4877dab285826e7c06744ed6a897aca367d3ffc4c641c08098c895a490baf744
BLAKE2b-256 checksum
How to use checksums
bb99f492dd3a8209449c2dbab492ff56607ec002934ea1db28700e1bb08866f0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Security for cleanedweb 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for cleanedweb-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page