Skip to main content

brightdata-hotdata-bridge

Collect web data with Bright Data scrapers and publish it into Hotdata tables, from Python or the bdh command line.

Give it a scraper's dataset id and a CSV of URLs (or keywords, usernames, and so on). It triggers the collection, waits for it, validates the records, and loads them into a Hotdata table.

inputs.csv ─► Bright Data trigger ─► poll until ready ─► download NDJSON
           ─► validate against the scraper schema ─► Hotdata upload ─► table load

Full documentation is in docs/: setup, configuration, usage, troubleshooting and architecture.

Features

  • Any Bright Data scraper. Dataset ids and their inputs are checked against Bright Data's published scraper catalog (1,200+ scrapers, cached locally), so a bad CSV is rejected before a paid collection starts.
  • Collect and discover. Supports collect_by_url and every discover_by_* method (keyword, category, user name, and more).
  • Validated before writing. Records are type-checked against the scraper's output schema. A mismatch pauses ingestion and writes a report. Records Bright Data flags as errors are set aside rather than loaded.
  • Large snapshots. The snapshot is streamed to disk and uploaded straight to Hotdata object storage, using multipart for big files. Loads run as background jobs.
  • Resumable and replayable. Every stage is saved to a local run record. A timeout, a failed write or a fixed schema problem is resumed with bdh replay, and a run is never loaded twice.
  • Polite to the APIs. Honours Retry-After with exponential backoff, and only retries a Bright Data trigger when it provably never reached the server.

Install

pip install brightdata-hotdata-bridge        # library + CLI
pipx install brightdata-hotdata-bridge       # CLI only, isolated environment

Requires Python 3.10+.

Configure

Copy .env.example to .env (or export the variables):

BRIGHTDATA_API_KEY=...
HOTDATA_API_KEY=...
HOTDATA_WORKSPACE_ID=...

The target database is looked up from the workspace before anything is triggered. If the workspace has exactly one database it is used automatically; if it has several, set HOTDATA_DATABASE_ID or HOTDATA_DATABASE_NAME (run bdh databases to list them). Each run records the database it used, so bdh replay always publishes to the same place.

Optional settings (schema, polling interval and timeout, state directory, catalog source) are documented in .env.example. The library and the CLI read the same configuration.

CLI

Find a scraper and the CSV columns it needs:

bdh scrapers list --search amazon
bdh scrapers show gd_l7q7dkf244hwjntr0

Collect and publish in one step:

# urls.csv has a header row:  url
bdh push --dataset-id gd_l7q7dkf244hwjntr0 --input-file urls.csv --table amazon_products

# A single required input can be passed inline
bdh push -d gd_l7q7dkf244hwjntr0 -q "https://www.amazon.com/dp/B0CRMZHDG8" -t amazon_products

# Discovery methods take their own inputs (keywords.csv header: keyword)
bdh push -d gd_l7q7dkf244hwjntr0 -m discover_by_keyword -i keywords.csv \
  --limit-per-input 50 -t amazon_search --mode append

Publish a snapshot that already exists, or start a collection and finish it later:

bdh push --snapshot-id sd_m1a2b3c4d5e6f7g8h --table amazon_products
bdh trigger -d gd_l7q7dkf244hwjntr0 -i urls.csv -t amazon_products   # prints snapshot id
bdh replay sd_m1a2b3c4d5e6f7g8h

Operate:

bdh status sd_m1a2b3c4d5e6f7g8h      # Bright Data status + local stage + last error
bdh runs                              # all local runs
bdh databases                         # Hotdata databases, and which one push will use
bdh replay sd_m1a2b3c4d5e6f7g8h       # resume after a timeout or failed write

Add --json to push, replay, status, runs and trigger for machine-readable output, and --log-json for JSON logs.

Load modes

Mode Effect
replace The snapshot becomes the table's contents (default).
append Rows are added to the existing table.
upsert Rows matching --key columns are replaced, others added.
update Rows matching --key columns are replaced.
delete Rows matching --key columns are removed.

replace is the default and discards every existing row. Pushing to a table that already holds data without --mode leaves only the new snapshot's rows. To keep earlier data, use --mode append, or --mode upsert --key <column> to keep one up-to-date row per record (for example --key id for LinkedIn profiles).

The first load into a new table is always replace, as Hotdata requires. If you ask for another mode on a table that does not exist yet, replace is used automatically. Every load is atomic: readers see the old contents or the new ones, never a mix, and a failed load leaves the table unchanged.

Validation options

  • --require-field name (repeatable): the field must be present and non-null.
  • --strict-fields: fail on fields that are not in the scraper's output schema (by default they are allowed and listed as schema drift in the report).
  • --stringify-nested: store nested objects and arrays as JSON text.

Exit codes

Code Meaning
0 Success
2 Configuration or usage error
3 Unknown dataset / method, or invalid input
4 Bright Data API error
5 Collection failed or was canceled
6 Snapshot not ready in time (resumable)
7 Schema mismatch or no publishable records
8 Hotdata write failed (replayable)
9 Local run record missing or unusable

Library

from brightdata_hotdata_bridge import push_snapshot_to_hotdata

result = push_snapshot_to_hotdata(
    dataset_id="gd_l7q7dkf244hwjntr0",
    input_file="urls.csv",
    table_name="amazon_products",
)
print(f"Loaded {result.row_count} rows; the table now has {result.table_row_count}")

examples/push_with_library.py shows how to handle each failure.

Async code (FastAPI, notebooks, orchestrators) should use the async API:

from brightdata_hotdata_bridge import (
    PushRequest,
    TableTarget,
    arun_pipeline,
    build_collection_request,
    get_settings,
)

settings = get_settings()
collection = await build_collection_request(
    dataset_id="gd_l7q7dkf244hwjntr0",
    queries=["https://www.amazon.com/dp/B0CRMZHDG8"],
    settings=settings,
)
result = await arun_pipeline(
    PushRequest(collection=collection, target=TableTarget(table="amazon_products")),
    settings,
)

Every error derives from BridgeError. Catch HotdataWriteError to schedule a replay_run(snapshot_id), or SchemaMismatchError to inspect exc.report_path.

How failures are handled

Situation Behaviour
Collection failed / canceled Logged with the snapshot id; nothing is written to Hotdata.
Snapshot not ready before timeout Run is saved; bdh push --snapshot-id or bdh replay resumes polling.
Schema or type mismatch Ingestion pauses; report.json lists every issue; nothing is written.
Hotdata upload or load fails Logged with Hotdata's trace id; bdh replay retries from that stage.
Bright Data / Hotdata rate limits Retry-After honoured, exponential backoff (2, 4, 8, 16, 32 s).

Run state lives in .bdh/runs/<snapshot_id>/: run.json (stage, target, upload id, last error), the raw and validated NDJSON, the rejected records, and the validation report.

Development

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
ruff check . && ruff format --check .
mypy
pytest --cov

See CONTRIBUTING.md and the Code of Conduct. Report security problems privately as described in SECURITY.md. Contributors sign the Contributor License Agreement on their first pull request. Licensed under Apache-2.0.

Metadata

Release files for brightdata-hotdata-bridge 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for brightdata-hotdata-bridge 0.1.1
File Size Uploaded Provenance Actions
brightdata_hotdata_bridge-0.1.1.tar.gz 64.9 kB With provenance Details for brightdata_hotdata_bridge-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for brightdata-hotdata-bridge 0.1.1
File Interpreter ABI Platform Actions
brightdata_hotdata_bridge-0.1.1-py3-none-any.whl Python 3 none any Details for brightdata_hotdata_bridge-0.1.1-py3-none-any.whl

Total release size: 117.0 kB

Release files / brightdata_hotdata_bridge-0.1.1.tar.gz

Download URL brightdata_hotdata_bridge-0.1.1.tar.gz
Size 64.9 kB
Tags Source
SHA-256 checksum
How to use checksums
efcd496f9464654406eacd58c067af720902300bc9e855c5396dfcc6b9fe29cb
BLAKE2b-256 checksum
How to use checksums
88104426c02457cf97b6efe2d1b6914397027b0e62d9220d6a3274d5b35363ee
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 20, 2026.

Transparency log

Release files / brightdata_hotdata_bridge-0.1.1-py3-none-any.whl

Download URL brightdata_hotdata_bridge-0.1.1-py3-none-any.whl
Size 52.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
fe9fcb7ceab0243bb35fcc6064f49b89ad9b08fa31917a16b51b5a430894770f
BLAKE2b-256 checksum
How to use checksums
55f32245a02363827e07833ef4997032828570ba4ca9c1af40f8f2ce282ff513
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 20, 2026.

Transparency log

Security for brightdata-hotdata-bridge 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for brightdata_hotdata_bridge-0.1.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 20, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page