langchain-scrapely

LangChain and LangGraph integration for Scrape.ly: run Actors, load their results as documents for RAG, and give agents Scrape.ly tools.

pip install langchain-scrapely
export SCRAPELY_TOKEN=your-api-key

Create an API key in the Scrape.ly dashboard under Settings → API keys. Everything below reads it from SCRAPELY_TOKEN, or takes it as scrapely_token=....

Load results as documents

ScrapelyWrapper runs an Actor, waits for it to finish, and returns a loader for its results. You decide how each result becomes a Document.

from langchain_core.documents import Document
from langchain_scrapely import ScrapelyWrapper

loader = ScrapelyWrapper().call_actor(
    actor_id='username/actor-name',
    run_input={'startUrls': [{'url': 'https://example.com'}]},
    dataset_mapping_function=lambda item: Document(
        page_content=item.get('text') or '', metadata={'source': item.get('url')}
    ),
)
docs = loader.load()

• call_actor_task(task_id, task_input, ...) runs a saved task instead. Fields in task_input override its saved input.
• acall_actor and acall_actor_task are the async versions.
• build, memory_mbytes and timeout_secs set the run options.
• A run that does not succeed raises RuntimeError.

To load results that already exist, use ScrapelyDatasetLoader:

from langchain_scrapely import ScrapelyDatasetLoader

loader = ScrapelyDatasetLoader(
    dataset_id='YOUR_DATASET_ID',
    dataset_mapping_function=lambda item: Document(page_content=item.get('text') or ''),
)
docs = loader.load()  # or lazy_load(), aload(), alazy_load()

Tools for agents

Any Actor as a tool

ScrapelyActorsTool turns one Actor into a tool. Its name, description and input come from the Actor, so the agent knows what it does and how to fill in its input.

from langchain_scrapely import ScrapelyActorsTool

tool = ScrapelyActorsTool('username/actor-name')
result = tool.invoke({'run_input': {'query': 'coffee shops in Berlin'}})

General tools

SCRAPELY_CORE_TOOLS holds all five as classes:

from langchain_scrapely import SCRAPELY_CORE_TOOLS

tools = [cls() for cls in SCRAPELY_CORE_TOOLS]

The tools that return results answer with JSON: {"run": {id, status, status_message, dataset_id, started_at, finished_at}, "items": [...]}. scrapely_run_actor and scrapely_run_task return only run, and scrapely_get_dataset_items returns "run": null. Errors, including invalid arguments, go back to the agent as text instead of stopping it.

Limits on what the agent may ask for, set when you create a tool:

With LangGraph

create_agent builds a LangGraph agent. It is in the langchain package, and the model below needs langchain-openai (pip install langchain langchain-openai):

from langchain.agents import create_agent
from langchain_scrapely import ScrapelyActorsTool, ScrapelyGetDatasetItemsTool

agent = create_agent(
    'openai:gpt-4o',
    tools=[ScrapelyActorsTool('username/actor-name'), ScrapelyGetDatasetItemsTool()],
)
agent.invoke({'messages': [{'role': 'user', 'content': 'Find the top coffee shops in Berlin'}]})

Give an agent only the tools it needs: fewer tools means better choices and fewer tokens.

Docs

scrape.ly/docs/langchain and scrape.ly/docs/langgraph.

License

MIT