ASIMOV Bright Data Module

ASIMOV module for data import powered by the Bright Data web data platform.

✨ Features

• Imports structured data from Airbnb, Amazon, Crunchbase, eBay, Facebook, Google, Indeed, Instagram, LinkedIn, Walmart, X (aka Twitter), Yahoo, and YouTube.
• Collects the raw JSON data via the Bright Data API (requires an API key).
• Constructs a semantic knowledge graph based on the KNOW ontology.
• Supports plain JSON output as well as RDF output in the form of JSON-LD.
• Distributed as a standalone static binary with zero runtime dependencies.

🛠️ Prerequisites

• Rust 1.85+ (2024 edition) if building from source code

⬇️ Installation

Installation from PyPI

pip install -U asimov-brightdata-module

Installation from RubyGems

gem install asimov-brightdata-module

Installation from NPM

npm install -g asimov-brightdata-module

Installation from Source Code

cargo install asimov-brightdata-module

👉 Examples

export BRIGHTDATA_API_KEY="..."

Fetching X Profiles

asimov-brightdata-fetcher https://x.com/bright_init   # JSON
asimov-brightdata-importer https://x.com/bright_init  # JSON-LD

Fetching LinkedIn Profiles

asimov-brightdata-fetcher https://www.linkedin.com/in/orlenchner/
asimov-brightdata-fetcher https://www.linkedin.com/company/bright-data/

Fetching Crunchbase Profiles

asimov-brightdata-fetcher https://www.crunchbase.com/organization/brightdata

Fetching Amazon Products

asimov-brightdata-fetcher https://www.amazon.com/Master-Algorithm-Ultimate-Learning-Machine/dp/0465094279

⚙ Configuration

Environment Variables

• BRIGHTDATA_API_KEY: (required) the Bright Data API key to use

📚 Reference

Installed Binaries

• asimov-brightdata-cataloger: discovers entities via the Bright Data API (not implemented yet)
• asimov-brightdata-fetcher: collects JSON data from the Bright Data API
• asimov-brightdata-importer: collects and transforms JSON into JSON-LD (not implemented yet)

Supported Datasets

Dataset	URL Prefix	JSON	RDF
Airbnb	https://www.airbnb.com/rooms/	✅	🚧
Amazon	https://www.amazon.com/	✅	🚧
	https://www.amazon.com/sp?seller=	✅	🚧
Crunchbase	https://www.crunchbase.com/organization/	✅	🚧
eBay	https://www.ebay.com/itm/	✅	🚧
Facebook	https://www.facebook.com/events/	✅	🚧
	https://www.facebook.com/groups/	✅	🚧
	https://www.facebook.com/marketplace/item/	✅	🚧
	https://www.facebook.com/share/p/	✅	🚧
Google	https://www.google.com/shopping/product/	✅	🚧
Indeed	https://www.indeed.com/cmp/	✅	🚧
Instagram	https://www.instagram.com/	✅	🚧
	https://www.instagram.com/p/	✅	🚧
	https://www.instagram.com/reel/	✅	🚧
LinkedIn	https://www.linkedin.com/company/	✅	🚧
	https://www.linkedin.com/in/	✅	🚧
	https://www.linkedin.com/jobs/	✅	🚧
	https://www.linkedin.com/posts/	✅	🚧
	https://www.linkedin.com/pulse/	✅	🚧
Walmart	https://www.walmart.com/global/seller/	✅	🚧
	https://www.walmart.com/ip/	✅	🚧
X (Twitter)	https://x.com/	✅	✅
Yahoo	https://finance.yahoo.com/quote/	✅	🚧
YouTube	https://www.youtube.com/@	✅	🚧
	https://www.youtube.com/watch?v=	✅	🚧

👨‍💻 Development

git clone https://github.com/asimov-modules/asimov-brightdata-module.git