Skip to main content

socialscrape-haystack

Haystack components and agent tools for SocialScrape, a social media data API: profiles, posts, comments, search results and video transcripts from YouTube, TikTok, Instagram, X, Facebook, LinkedIn, Reddit and more, as JSON in one response shape, with one API key.

pip install -U socialscrape-haystack
export SOCIALSCRAPE_API_KEY="ss_live_..."   # get a key: https://socialscrape.dev/sign-up

Python 3.11 or newer, Haystack 2.14 or newer. The package builds on the official socialscrape-sdk client, so retries, timeouts and long transcription jobs are handled for you. Full guide: socialscrape.dev/docs/haystack.

Components

Class What it does
SocialScrapeTranscriptsFetcher Video URLs in, one transcript Document per URL out. A URL that fails goes to errors, not an exception.
SocialScrapeFetcher Runs any endpoint by id with its parameters and returns Documents: one per item of a list, one otherwise.
SocialScrapeToolset Eight tools for a Haystack Agent: profile, posts, one post, comments, transcript, search, endpoint lookup and call-any-endpoint.

The API key is a Haystack Secret. The default is Secret.from_env_var("SOCIALSCRAPE_API_KEY"), and every component serializes with to_dict and from_dict without writing the key out.

Transcripts into a pipeline

from haystack import Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.components.preprocessors import DocumentSplitter
from haystack.dataclasses import ChatMessage
from socialscrape_haystack import SocialScrapeTranscriptsFetcher

template = [ChatMessage.from_user(
    "Summarize the main points of these transcripts.\n"
    "{% for doc in documents %}\n[{{ doc.meta.url }}]\n{{ doc.content }}\n{% endfor %}"
)]

pipe = Pipeline()
pipe.add_component("fetcher", SocialScrapeTranscriptsFetcher())
pipe.add_component("splitter", DocumentSplitter(split_by="word", split_length=300, split_overlap=30))
pipe.add_component("prompt", ChatPromptBuilder(template=template, required_variables=["documents"]))
pipe.add_component("llm", OpenAIChatGenerator(model="gpt-4o-mini"))
pipe.connect("fetcher.documents", "splitter.documents")
pipe.connect("splitter.documents", "prompt.documents")
pipe.connect("prompt.prompt", "llm.messages")

result = pipe.run({"fetcher": {"urls": [
    "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
    "https://www.tiktok.com/@nasa/video/7665075736742530317",
]}})
print(result["llm"]["replies"][0].text)

For retrieval, put an embedder and a DocumentWriter after the splitter instead of the prompt builder. Each chunk keeps the meta of its video (url, platform, language, duration_seconds).

A transcript Document has the transcript text as content and this meta: url and source (the video URL), platform, video_id, language, duration_seconds, transcript_source (captions, or ai when the audio was transcribed) and endpoint.

Errors

SocialScrapeTranscriptsFetcher does not fail a batch for one bad link. A URL that cannot be fetched (private video, unsupported link, no captions) is left out of documents and listed in errors:

result = SocialScrapeTranscriptsFetcher().run(urls=["https://www.youtube.com/watch?v=dQw4w9WgXcQ", "https://example.com/nope"])
result["errors"]   # [{"url": "https://example.com/nope", "error": "..."}]

A failed request costs no credits. Pass raise_on_failure=True to raise on the first failure instead. A video with no captions is transcribed from its audio up to 2 minutes; longer audio needs allow_long_transcription=True, which is charged per started minute (see Billing).

Any endpoint as Documents

from socialscrape_haystack import SocialScrapeFetcher

fetcher = SocialScrapeFetcher(max_pages=3)

# One Document per comment, up to three pages
comments = fetcher.run(
    endpoint="youtube.video.comments",
    params={"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"},
)["documents"]

# One Document for a profile
profile = fetcher.run(endpoint="tiktok.profile", params={"handle": "nasa"})["documents"][0]
profile.content   # name, headline and bio
profile.meta      # {'platform': 'tiktok', 'handle': 'nasa', 'followers': ..., 'endpoint': 'tiktok.profile', ...}

You can also fix the endpoint and its parameters when you create the component and pass only what changes at run time: SocialScrapeFetcher(endpoint="reddit.search", params={"sort": "top"}).run(params={"query": "mechanical keyboards"}).

  • A list endpoint gives one Document per item and is followed across pages up to max_pages (default 1; each page is one request). max_items stops early.
  • Any other endpoint (a profile, one post, a transcript) gives one Document.
  • The content is the item's text: a post's title and text, a comment's text, a transcript's text, a profile's name and bio. An object with no text of its own is kept whole as JSON.
  • The item's other scalar fields, its counts and its author go to meta, flat, with endpoint, platform and the item's url (also as source).
  • By default a refused request raises. With raise_on_failure=False it is returned in errors.

The endpoint ids and their parameters are in the API reference, and in code:

from socialscrape_haystack import endpoints

endpoints()["tiktok.profile.videos"].signature()

Parameters are checked against the endpoint before the request is made, so a misspelt name is answered at once and costs nothing.

Give an Agent the tools

from haystack.components.agents import Agent
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from socialscrape_haystack import SocialScrapeToolset

agent = Agent(
    chat_generator=OpenAIChatGenerator(model="gpt-4o-mini"),
    tools=SocialScrapeToolset(),   # reads SOCIALSCRAPE_API_KEY
    system_prompt="You answer questions about social media accounts and videos. Use the tools; do not guess.",
)

result = agent.run(messages=[ChatMessage.from_user(
    "What did NASA post on YouTube lately, and what are people saying in the comments?"
)])
print(result["last_message"].text)

Any chat generator that supports tool calling works. The toolset has eight tools that share one client:

Tool What it does
socialscrape_get_profile A profile, channel or artist: name, bio, links, verification, follower counts.
socialscrape_list_posts The latest posts or videos of a profile, channel or page, one page per call.
socialscrape_get_post One post, video, tweet, pin or clip from its URL.
socialscrape_list_comments The comments on a post or video, one page per call.
socialscrape_get_transcript The transcript of a video from its URL.
socialscrape_search Keyword search on one platform, one page per call.
socialscrape_list_endpoints What else exists: every endpoint id with its parameters and price. Makes no request, so it is free.
socialscrape_call_endpoint Runs any endpoint by id: ad libraries, followers, hashtags, playlists, music, shops and the rest.

A tool answers with a JSON string: data (the object, or a list page with items, cursor and has_more), credits_charged and credits_remaining. A request the API refuses is free and comes back as the tool's answer (error: not_found: ...), so the agent can read it and try something else. Tools that list things return a cursor to pass back for the next page.

Toolset options: api_key, max_age_days (accept a cached result up to this many days old, which costs 0 credits), trim (default True, a smaller response that saves context), client and async_client (a socialscrape.SocialScrape or AsyncSocialScrape you configured yourself) and base_url.

Billing

Most requests cost 1 credit: a profile, a post, a page of a list or of search results, a transcript from captions. AI transcription of a video with no captions costs 1 credit up to 2 minutes of audio; longer audio is transcribed only when you pass allow_long_transcription, at 5 credits per started minute of the whole video. A failed request and a cached repeat cost 0. A new account starts with 100 free credits, no card needed. See pricing.

Tests

pip install -e ".[test]"
pytest

The tests run offline against sample responses from the API reference.

License

MIT, the same license as langchain-socialscrape.

Metadata

Release files for socialscrape-haystack 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for socialscrape-haystack 0.1.1
File Size Uploaded Provenance Actions
socialscrape_haystack-0.1.1.tar.gz 62.5 kB Without provenance Details for socialscrape_haystack-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for socialscrape-haystack 0.1.1
File Interpreter ABI Platform Actions
socialscrape_haystack-0.1.1-py3-none-any.whl Python 3 none any Details for socialscrape_haystack-0.1.1-py3-none-any.whl

Total release size: 119.9 kB

Release files / socialscrape_haystack-0.1.1.tar.gz

Download URL socialscrape_haystack-0.1.1.tar.gz
Size 62.5 kB
Tags Source
SHA-256 checksum
How to use checksums
167b59ab7d906e4a123aec9436b867359aa6072b54537ce934c3791916688e78
BLAKE2b-256 checksum
How to use checksums
63ca6dadb83bdfc94a42d10c0a8c76375073a05525884f4f92a9ede755a8abcb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Release files / socialscrape_haystack-0.1.1-py3-none-any.whl

Download URL socialscrape_haystack-0.1.1-py3-none-any.whl
Size 57.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
0dd0ab35b8b7f8914812fc107a936984464482d435698dc582f81ee7833be78a
BLAKE2b-256 checksum
How to use checksums
2482b4d5a91cfd974f7c3061a9afab614db21cba4a095ac0ad2ed587521be1f5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Security for socialscrape-haystack 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page