Skip to main content

crewai-socialscrape

CrewAI tools for SocialScrape, a social media data API: profiles, posts, comments, search results and video transcripts from YouTube, TikTok, Instagram, X, Facebook, LinkedIn, Reddit and more, as JSON in one response shape, with one API key.

pip install -U crewai-socialscrape
export SOCIALSCRAPE_API_KEY="ss_live_..."   # get a key: https://socialscrape.dev/app

Python 3.11 or newer. The package builds on the official socialscrape-sdk client, so retries, timeouts and long transcription jobs are handled for you. Guide: socialscrape.dev/docs/crewai.

Give an agent the tools

from crewai import Agent, Crew, Task
from crewai_socialscrape import get_socialscrape_tools

analyst = Agent(
    role="Social media analyst",
    goal="Answer questions about creators and what their audiences say",
    backstory="You look at the data before you answer and you say where each number came from.",
    tools=get_socialscrape_tools(),   # reads SOCIALSCRAPE_API_KEY
)

task = Task(
    description=(
        "Compare NASA's TikTok and YouTube channels: followers, the latest posts, "
        "and what people say in the comments of the newest YouTube video."
    ),
    expected_output="A short comparison with the numbers and three points from the comments.",
    agent=analyst,
)

result = Crew(agents=[analyst], tasks=[task]).kickoff()
print(result.raw)

get_socialscrape_tools() returns eight tools that share one client:

Tool What it does Credits
SocialScrapeProfileTool (socialscrape_get_profile) A profile, channel or artist: name, bio, links, verification, follower counts. 17 platforms. 1
SocialScrapePostsTool (socialscrape_list_posts) The latest posts or videos of a profile, channel or page, one page per call. 14 platforms. 1 per page
SocialScrapePostTool (socialscrape_get_post) One post, video, tweet, pin or clip from its URL. 18 platforms. 1
SocialScrapeCommentsTool (socialscrape_list_comments) The comments on a post or video, one page per call. 7 platforms. 1 per page
SocialScrapeTranscriptTool (socialscrape_get_transcript) The transcript of a video from its URL. 1 from captions; see below
SocialScrapeSearchTool (socialscrape_search) Keyword search on one platform, one page per call. 14 platforms. 1 per page
SocialScrapeEndpointsTool (socialscrape_list_endpoints) What else exists: every endpoint id with its parameters and price. Makes no request. free
SocialScrapeCallEndpointTool (socialscrape_call_endpoint) Runs any of the 193 endpoints by id: ad libraries, followers, hashtags, playlists, music, shops and the rest. per endpoint, mostly 1

The platform is an argument (platform="tiktok", handle="nasa"), and for a post, its comments or a transcript it is read from the link. Tools that list things return a cursor to pass back for the next page.

Use one tool directly

from crewai_socialscrape import SocialScrapeProfileTool, SocialScrapeTranscriptTool

profile = SocialScrapeProfileTool()            # reads SOCIALSCRAPE_API_KEY
profile.run(platform="tiktok", handle="nasa")
# '{"data": {"object": "profile", "platform": "tiktok", "handle": "nasa", "name": "NASA", ...,
#   "stats": {"followers": ..., "following": ..., "posts": ..., "likes": ...}},
#   "credits_charged": 1, "credits_remaining": 4999}'

SocialScrapeTranscriptTool().run(url="https://www.youtube.com/watch?v=dQw4w9WgXcQ")
# '{"data": {"object": "transcript", "language": "en", "source": "captions", "text": "..."}, ...}'

Every tool answers with a JSON string: data (the object, or a list page with items, cursor and has_more), credits_charged and credits_remaining. The transcript tool leaves out the timed segments unless you pass include_segments=True.

Any endpoint, by id:

from crewai_socialscrape import SocialScrapeCallEndpointTool, SocialScrapeEndpointsTool

SocialScrapeEndpointsTool().run(search="ad library")        # which endpoints, what they take
SocialScrapeCallEndpointTool().run(
    endpoint="facebook.ad_library.search",
    params={"query": "running shoes"},
)

Parameters are checked against the endpoint before the request is made, so a misspelt name is answered at once and costs nothing. Only endpoints in the shipped catalogue can be called.

Errors

A request the API refuses (no such account, a private post, a video with no transcript) is free. The tool returns the error as its answer, for example Error: not_found: No TikTok account with that handle. | HTTP 404 | request req_..., so an agent can read it and try something else. A missing API key raises MissingApiKeyError at the first call.

Options

Each tool, and get_socialscrape_tools(), takes:

  • api_key: the SocialScrape key. Left out, it is read from SOCIALSCRAPE_API_KEY.
  • max_age_days: accept a cached result up to this many days old (0 to 30). A cached result costs 0 credits.
  • trim (default True): ask for the smaller response, without null fields and the extra object, which saves context.
  • client / async_client: a socialscrape.SocialScrape or AsyncSocialScrape you configured yourself (timeouts, retries, your own httpx client).
  • base_url: API base URL override.

The tools work with run and arun.

Billing

Most requests cost 1 credit: a profile, a post, a page of a list or of search results, a transcript from captions. AI transcription of a video with no captions costs 1 credit up to 2 minutes of audio; longer audio is transcribed only when you pass allow_long_transcription=True, at 5 credits per started minute of the whole video. A failed request and a cached repeat cost 0. A new account starts with 100 free credits, no card needed. See pricing.

Because an agent decides how many calls to make, set a limit on a tool with CrewAI's max_usage_count if you want a hard cap on a run.

Tests

pip install -e ".[test]"
pytest

The tests run offline against sample responses from the API reference.

License

MIT

Metadata

Release files for crewai-socialscrape 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for crewai-socialscrape 0.1.0
File Size Uploaded Provenance Actions
crewai_socialscrape-0.1.0.tar.gz 58.8 kB Without provenance Details for crewai_socialscrape-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for crewai-socialscrape 0.1.0
File Interpreter ABI Platform Actions
crewai_socialscrape-0.1.0-py3-none-any.whl Python 3 none any Details for crewai_socialscrape-0.1.0-py3-none-any.whl

Total release size: 109.4 kB

Release files / crewai_socialscrape-0.1.0.tar.gz

Download URL crewai_socialscrape-0.1.0.tar.gz
Size 58.8 kB
Tags Source
SHA-256 checksum
How to use checksums
dddf27c5b2681784e79287ccbbf18b8e14046f3bfe71afeecd392ba7f54551f0
BLAKE2b-256 checksum
How to use checksums
a2cb8b0067851fd2e97a40266003e77df24a9eb8102414269252c4908c178342
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Release files / crewai_socialscrape-0.1.0-py3-none-any.whl

Download URL crewai_socialscrape-0.1.0-py3-none-any.whl
Size 50.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
20b0a80120bfb360e4e80781928d57b545c335dec3432a84797a8d7041f6b89b
BLAKE2b-256 checksum
How to use checksums
52b95c907339222da1f7b8451ab1db49949ce5ae0cd794be8260105950d47000
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Security for crewai-socialscrape 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page