llama-index-readers-scrapely

LlamaIndex readers for Scrape.ly: run an Actor, or read an existing dataset, and get Documents back for your index.

pip install llama-index-readers-scrapely
export SCRAPELY_TOKEN=your-api-key

Create an API key in the Scrape.ly dashboard under Settings → API keys. The readers read it from SCRAPELY_TOKEN, or take it as scrapely_token=....

Run an Actor

from llama_index.core import Document, VectorStoreIndex
from llama_index.readers.scrapely import ScrapelyActor

docs = ScrapelyActor().load_data(
    actor_id='username/actor-name',
    run_input={'startUrls': [{'url': 'https://example.com'}]},
    dataset_mapping_function=lambda item: Document(
        text=item.get('text') or '', metadata={'url': item.get('url')}
    ),
)
index = VectorStoreIndex.from_documents(docs)

The index and query engine use OpenAI by default: pip install llama-index (or llama-index-embeddings-openai llama-index-llms-openai) and set OPENAI_API_KEY.

build, memory_mbytes and timeout_secs set the run options. A run that does not succeed raises RuntimeError.

Read an existing dataset

from llama_index.readers.scrapely import ScrapelyDataset

docs = ScrapelyDataset().load_data(
    dataset_id='YOUR_DATASET_ID',
    dataset_mapping_function=lambda item: Document(text=item.get('text') or ''),
)

dataset_mapping_function decides how each result becomes a Document, and must return one.

Docs

scrape.ly/docs/llama-index

License

MIT