langchain-socialscrape

LangChain tools and a document loader for SocialScrape, a social media data API: profiles, posts, comments, search results and video transcripts from YouTube, TikTok, Instagram, X, Facebook, LinkedIn, Reddit and more, as JSON in one response shape, with one API key.

pip install -U langchain-socialscrape
export SOCIALSCRAPE_API_KEY="ss_live_..."   # get a key: https://socialscrape.dev/sign-up

Python 3.11 or newer. The package builds on the official socialscrape-sdk client, so retries, timeouts and long transcription jobs are handled for you.

Give an agent the tools

from langchain.agents import create_agent
from langchain_socialscrape import SocialScrapeToolkit

tools = SocialScrapeToolkit().get_tools()   # reads SOCIALSCRAPE_API_KEY
agent = create_agent("openai:gpt-5", tools=tools)

result = agent.invoke({"messages": [
    ("user", "What did NASA post on YouTube this week, and what are people saying in the comments?")
]})
print(result["messages"][-1].content)

get_tools() returns eight tools that share one client:

The platform is an argument ({"platform": "tiktok", "handle": "nasa"}), and for a post, its comments or a transcript it is read from the link. Tools that list things return a cursor to pass back for the next page.

Use one tool directly

from langchain_socialscrape import SocialScrapeProfileTool, SocialScrapeTranscriptTool

SocialScrapeProfileTool().invoke({"platform": "tiktok", "handle": "nasa"})
# '{"data": {"object": "profile", "platform": "tiktok", "handle": "nasa", "name": "NASA", ...,
#   "stats": {"followers": ..., "following": ..., "posts": ..., "likes": ...}},
#   "credits_charged": 1, "credits_remaining": 4999}'

SocialScrapeTranscriptTool().invoke({"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"})
# '{"data": {"object": "transcript", "language": "en", "source": "captions", "text": "..."}, ...}'

Every tool answers with a JSON string: data (the object, or a list page with items, cursor and has_more), credits_charged and credits_remaining.

Any endpoint, by id:

from langchain_socialscrape import SocialScrapeCallEndpointTool, SocialScrapeEndpointsTool

SocialScrapeEndpointsTool().invoke({"search": "ad library"})        # which endpoints, what they take
SocialScrapeCallEndpointTool().invoke({
    "endpoint": "facebook.ad_library.search",
    "params": {"query": "running shoes"},
})

Parameters are checked against the endpoint before the request is made, so a misspelt name is answered at once and costs nothing.

Errors

A request the API refuses (no such account, a private post, a video with no transcript) is free. By default the tool returns the error as its answer, for example not_found: No TikTok account with that handle. | HTTP 404 | request req_..., so an agent can read it and try something else. Pass handle_tool_error=False to have the tool raise ToolException instead. A missing API key always raises.

Options

Each tool, and the toolkit, takes:

• api_key: the SocialScrape key. Left out, it is read from SOCIALSCRAPE_API_KEY.
• max_age_days: accept a cached result up to this many days old (0 to 30). A cached result costs 0 credits.
• trim (default True): ask for the smaller response, without null fields and the extra object, which saves context.
• client / async_client: a socialscrape.SocialScrape or AsyncSocialScrape you configured yourself (timeouts, retries, your own httpx client).
• base_url: API base URL override.

The tools work with invoke and ainvoke.

Load documents for RAG

SocialScrapeLoader turns the result of any endpoint into LangChain Documents.

from langchain_socialscrape import SocialScrapeLoader

# One Document per comment, three pages of them
docs = SocialScrapeLoader(
    "youtube.video.comments",
    {"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"},
    max_pages=3,
).load()

docs[0].page_content   # the comment text
docs[0].metadata       # {'source': 'https://www.youtube.com/watch?v=...&lc=...', 'author': '...',
                       #  'published_at': '...', 'likes': 12, 'platform': 'youtube', ...}

# A transcript as one Document
transcript = SocialScrapeLoader("transcript", {"url": "https://www.tiktok.com/@nasa/video/7300000000000000000"}).load()

# A subreddit search, ready for a vector store
posts = SocialScrapeLoader("reddit.search", {"query": "mechanical keyboards", "sort": "top"}, max_items=50, max_pages=3).load()

• A list endpoint gives one Document per item and is followed across pages up to max_pages (default 1; each page is one request). max_items stops early.
• Any other endpoint (a profile, one post, a transcript) gives one Document.
• The page content is the item's text: a post's title and text, a comment's text, a transcript's text, a profile's name and bio. An object with no text of its own is kept whole as JSON.
• The item's other scalar fields, its counts (views, likes, comments, ...) and its author go to metadata, flat, with the item's URL as source.
• lazy_load() yields Documents as each page arrives.

The endpoint ids and their parameters are in the API reference, and in code:

from langchain_socialscrape import endpoints

endpoints()["tiktok.profile.videos"].signature()   # 'handle | id (one needed), sort, include_pinned, cursor, region'

Billing

Most requests cost 1 credit: a profile, a post, a page of a list or of search results, a transcript from captions. AI transcription of a video with no captions costs 1 credit up to 2 minutes of audio; longer audio is transcribed only when you pass allow_long_transcription, at 5 credits per started minute of the whole video. A failed request and a cached repeat cost 0. A new account starts with 100 free credits, no card needed. See pricing.

Tests

pip install -e ".[test]"
pytest

The tests run offline against sample responses from the API reference, and include LangChain's standard unit tests for tools.

License

MIT