matrx-social

Provider-agnostic social data for Social Intelligence (common-docs/systems/marketing/social/). Depends only on pydantic + httpx; owns no ledger, no DB, no mock mode.

• Contracts (contracts.py): Platform, SocialProfile, SocialPost, MetricSet, Comment, Transcript, Ad, Page[T], ProviderResult[T]. A metric the provider does not give is None, never 0. Every result carries provider, fallback_reason, cost_credits, and the raw payload.
• Interface (interface.py): SocialProvider — get_post, get_profile, list_posts, get_comments, get_transcript, search, trending, search_ads, capabilities.
• ScrapeCreators (providers/scrapecreators/): primary. Only operations whose real payload was captured and normalized are in capabilities(); anything else raises UnsupportedError. Snapchat and comments/search on several platforms are NOT wired yet (see common-docs/systems/marketing/social/STATE.md for the measured picture). A lookup of a missing account returns HTTP 200 + account_deactivated and is billed 1 credit; the adapter raises NotFoundError(credits_charged=1). A lookup the platform refused answers HTTP 200 + success: true with error/errorStatus/message (e.g. Instagram not_found 404 "Profile is private"); _get raises NotFoundError for it (other errorStatus values: SocialProviderError, retryable only on 5xx) — never an empty normalized profile.
• Pinterest (wired 2026-10-09): no profile endpoint exists. get_profile reads /v1/pinterest/user/boards (1 credit; the owner block gives handle, id, followers; extras.boards holds the boards on that page; post_count stays None unless the boards listing is complete). A board URL resolves to its owner. list_posts = pins of the account's boards, ONE board per page (cursor = board index, 1 credit/page, boards with 0 pins skipped; effective_params names the board). get_post = /v1/pinterest/pin; keyword search = /v1/pinterest/search (items carry NO engagement numbers: all metrics None; non-pin cards skipped and counted in effective_params.skipped_non_pins). No comments, no transcript. Outlier metric = saves (aggregatedStats.saves, the platform's own measure, larger than repinCount which rides in extras.repins); likes = reaction total, comments = commentCount, shares = shareCount, views always None. Pin video shape was not captured (no video pin in any probe): a video pin without a parsable mp4 carries extras.video_unparsed.
• Reddit (wired 2026-10-09): no user endpoint, so the trackable unit is a SUBREDDIT (handle = subreddit name, display_name = r/name, id = t5_…). get_profile = /v1/reddit/subreddit/details (followers = subscribers, None when the vendor sends null, as it did for r/Entrepreneur); list_posts = /v1/reddit/subreddit (sort=new, after cursor); get_post = /v1/reddit/post; get_comments = /v1/reddit/post/comments (cursor from more.cursor); keyword search = /v1/reddit/search (posts). Outlier metric = score (net upvotes) stored as likes; comments = num_comments; views None. No transcript wired (the vendor has one for video posts).
• Threads now also has keyword search (/v1/threads/search, one page, no cursor).
• Outlier metric per platform lives in aidream/services/social/stats.py::PRIMARY_METRIC (views; Pinterest saves; Reddit and Threads likes) and is mirrored in matrx-frontend features/marketing/social/outlier.ts. The stored post_stat.baseline_views column holds the baseline of that platform's primary metric; velocity_24h is only computed for views platforms.
• Transcripts (normalize.parse_transcript): captions are cleaned (clean_caption_text: zero-width characters stripped, whitespace collapsed) and rolling/overlapping segments de-duplicated (dedupe_caption_segments); text is rebuilt from the segments, never taken from the vendor's transcript_only_text (YouTube repeats every line there: a 20-minute video counted 54,558 words, really ~11k). Test on the real payload tests/fixtures/yt_transcript_rolling.json.
• HTTP content refusals: a structured HTTP 403 verdict naming a restricted/private/missing post or profile is classified by the same errors.refusal_kind as a 200 refusal. Unknown 403s and credential/IP refusals remain AuthError; zero stated credits stay zero. Regression fixture: tests/fixtures/ig_post_age_restricted.json.
• Mastodon (providers/mastodon_public/, MKT-109): free, keyless, always configured. An account is read from ITS OWN server (/api/v1/accounts/lookup, /accounts/<id>/statuses); handles are user@host (a bare user is mastodon.social); profile fields ride in extras["fields"], the first (verified first) link in extras["external_url"]. Cost is always 0. A host that does not answer the Mastodon API is NotFoundError(reason="missing").
• Failure types: a vendor payload missing a key the adapter needs raises UnexpectedShapeError(endpoint, missing) (the caller records it and may fall back). AuthError/OutOfCreditsError are the account-level failures a caller treats as critical (ProviderAccountError is what a caller raises when every provider fails that way). Defaults the adapter applied (trending region, Meta ads country) are reported on ProviderResult.effective_params.
• Routing is not in this package. The provider router (chain order, fallback, chat-vs-screen read_context, definitive-refusal stop, credit aggregation) was deleted in 0.1.1; routing now lives as data in the platform tool catalog (aidream/services/tool_catalog/seeds/social.py, tested by aidream/services/tool_catalog/tests/test_social_family_is_the_old_router_as_data.py). This package keeps only the single-provider adapters, the contracts (ReadContext and DEEPER_OPTION are in contracts.py) and the error types. list_posts(profile, cursor, limit=N) carries the TOTAL posts wanted (a hint for count-based providers; Bright Data deep posts read it, ScrapeCreators ignores it). liveness.check_profile(paid=True, reader=…) takes the host's paid reader (a get_profile that runs the host's catalog job); with no reader a paid check answers unknown and spends nothing, and vendor_says_missing(platform, detail) lets a host's adapter name the platform's own 404 "does not exist". EnsembleData (ENSEMBLEDATA_TOKEN) and Apify (APIFY_API_TOKEN) are registered in providers/__init__.py but have no key in any .env and no adapter yet.
• Cost: transport.py calls registered observers with SocialBillableCall(credits_charged). aidream wires aidream/services/runtime/external_api_observers.py::_on_social_call (units = credits, priced by knob platform.external_api_prices.scrapecreators_credit_usd).
• Key: env SCRAPECREATORS_API_KEY (REQUIRED_ENV, credential definition scrapecreators_api_key).

Tests: pytest packages/matrx-social -m "not live" — unit tests normalize REAL captured payloads (tests/fixtures/, captured 2026-10-09; Pinterest/Reddit/Threads-search fixtures pin_*, reddit_*, th_search); -m live makes real paid calls (~12 credits, skipped without the key; a plain pytest WITH the key set runs them and spends). Battery: scripts/day_one_battery.py OUT.json (~60 credits).

The raw-answer seam (Watch AI RAW tier, 2026-10-22)

register_social_response_observer(fn) — fn(SocialVendorResponse) is told of EVERY ScrapeCreators response exactly as received: body is response.content (bytes, never a re-serialized decode), every attempt (retries included), failures and unbilled answers, billed as the vendor states it, transport errors (transport_error, empty body) and test-replay hits (replayed). It is separate from the billing observer (SocialBillableCall), which stays a cost record. The host (aidream services/ai_watch/raw_capture_sink.py) puts each answer on the running step's capture scope. An observer that raises is logged, never the call's failure. Tests: tests/test_vendor_response_observer.py. Bright Data and grok_x answers are not on the seam yet.

grok_x — X data through Grok (second X route, beside the direct X API)

providers/grok_x/client.py; env XAI_API_KEY (already in REQUIRED_ENV); optional GROK_X_MODEL (default grok-4.20-0309-non-reasoning, the cheapest model measured to work; never a premium/reasoning model). Calls the xAI Responses API (POST /v1/responses) with the server-side x_search tool, max_turns=2, and a strict JSON schema — one request, no agent loop.

• Supports on X only: get_post (status URL), get_profile, list_posts (by handle), keyword + hashtag search. Everything else raises UnsupportedError. No cursor: one page of max_posts (default 10; also the spend cap).
• Registered in providers/__init__.py as grok_x after ScrapeCreators and Bright Data. Order between providers is the catalog's, not this package's.
• Honesty: xAI does not hand back the raw X data; numbers reach us transcribed by the model. Every item carries extras["extraction"]="grok_transcribed"; not-shown fields are None (profile following/post_count came back null on NASA); posts with a non-numeric id are dropped (effective_params.dropped_invalid). Metrics differ slightly between two reads seconds apart (live counts) — treat as approximate and prefer ScrapeCreators for get_post/profile/list when both exist.
• Cost: xAI reports exact USD per request (usage.cost_in_usd_ticks, 1 tick = 1e-10 USD; tokens + X Search at $5/1k posts and $10/1k profiles fetched, per docs.x.ai/developers/tools/x-search and /pricing). Surfaced as ProviderResult.cost_usd and SocialBillableCall.reported_usd; _on_social_call records ONE external_api ledger row priced by the vendor figure (no knob). It bypasses matrx-ai, so no token row exists: counted exactly once. SOC-POINTS reads that row (provider=grok_x). Measured: profile ~$0.033, 3 posts ~$0.02-0.04, search ~$0.04.
• Ruling (AI-mandate law): this is a vendor data-retrieval call with a fixed extraction prompt and no persona, tools choice, or loop — an API adapter like ScrapeCreators, not an agent, so it does not go through a mandate/agent factory. If a future use needs Grok to reason over results, that is a separate mandate-backed agent fed by this adapter's output.
• Rate limits: not published for x_search ("per-minute limits apply"); retried on 429/5xx with Retry-After. Terms of X-content use via xAI not reviewed — open.
• Tests: tests/test_grok_x.py (real captured payload fixtures/grok_x_responses_nasa_latest.json); live test_grok_x_live_search_tiny_spend (~$0.02).

brightdata — second provider (fallback behind ScrapeCreators)

providers/brightdata/{client,normalize}.py; env BRIGHT_DATA_API_KEY (read like SCRAPECREATORS_API_KEY; already in REQUIRED_ENV). Bright Data Web Scraper API datasets, billed per returned record.

• Wire: POST /datasets/v3/scrape?dataset_id=…&format=json&include_errors=true, Bearer key, body [{"url": …}]. Sync 200 (records) within ~60 s, else 202 {snapshot_id} → poll /datasets/v3/progress/{id} (bounded: poll_timeout_s=180, announced in the error; a timed-out job is recorded at its cap, never as free) → /datasets/v3/snapshot/{id}. A read timeout on the POST is never retried (the job may already bill). Error records: dead_page → NotFoundError(reason="missing"); blocked/crawl_error → retryable SocialProviderError.
• 🚨 Cap: every request carries limit_per_input AND limit_multiple_results; BrightDataTransport.scrape raises UncappedRequestError before sending without a positive cap (an uncapped IG comments read returned 522 records). Timeout: a snapshot poll timeout POSTs /datasets/v3/snapshot/{id}/cancel (never retried), reports "job cancelled" or "cancel FAILED" in the error and the call detail, and still books the cap as spend. No double jobs: the trigger POST is retried only on connection errors before send and on 429 with Retry-After; a 5xx/read timeout is never re-POSTed (it may have started a billed job). Live 2026-10-11: both params at 2 → exactly 2 IG comment records. Which one alone is honored was not isolated (testing the wrong one alone could bill hundreds of records).
• Cost: SocialBillableCall(provider="brightdata", credits_charged=records returned), path /brightdata/<dataset_id>/<op> with the operation word (profile/post/posts/comments) so _on_social_call classifies it; priced by knob platform.external_api_prices.brightdata_record_usd (0.0015). Error records count as returned records (conservative; unverified whether Bright Data bills them).
• Why its own transport, not matrx-scraper's Bright Data client: a social read must be costed through the social observer (_on_social_call): subject attribution (cost_subject) and the shared-refresh cost split across tracking organizations. matrx-scraper's client feeds _on_brightdata_call (site data, no subject, no split). Same vendor, two cost meanings.
• Capabilities (only what a real record was normalized + tested for; tests/test_brightdata.py, fixtures tests/fixtures/brightdata/): Instagram profile/post/list/comments; TikTok profile/post/list/comments (comments embedded in the post record); YouTube channel/video; LinkedIn person/post/list/comments (embedded top_visible_comments); Facebook page profile; X profile/post/list; Threads profile; Pinterest profile; Reddit list (subreddit discover, cap = posts_page_size 25). Everything else (transcripts, search, trending, ads, cursors) raises UnsupportedError. One page only: a cursor is refused, never re-billed.
• Deep posts (X, LinkedIn): list_posts uses TRIGGER mode, POST /datasets/v3/trigger?dataset_id=…&type=discover_new&discover_by=profile_url (the sync /scrape returned 0 records for a discover), always snapshot-polled with its own bound BrightDataTransport(discover_timeout_s=600) (cancel-on-timeout, cap booked, fail-closed cap unchanged). X input carries num_of_posts = the requested count; the LinkedIn posts dataset REJECTS num_of_posts (400), so only limit_per_input bounds it. Count = limit hint, else BrightDataProvider(deep_posts_default=50), ceiling deep_posts_max=200; one page, no cursor. Measured: 50 X posts in 27 s, 10 LinkedIn posts with like/comment counts in 34 s. Other platforms: the next sentence.
• list_posts (other platforms) reads the posts embedded in the profile record; a profile record fetched in the last 10 minutes is reused (0 records). Instagram and LinkedIn embedded posts carry NO metrics (all None).
• Honesty rules: X profiles answer followers/following/posts_count = 0 beside real posts → all None, extras.counts_withheld. X retweets in a profile carry likes=0, replies=0 → None. Ids follow ScrapeCreators (IG pk, Reddit id without t3_, LinkedIn no platform_user_id) so both providers land on the same rows. Every item has extras.source_dataset; raw = the record.
• Live proof 2026-10-11: local server with a dead ScrapeCreators key → /social/ingest/profile TikTok @nasa fell back (fallback_reason "scrapecreators error: … HTTP 401"), stored 2,100,000 followers / 56 posts / 10.9M likes + 17 posts, 1 record = $0.0015 on one external_api row.

Data defects fixed 2026-10-11 (head-to-head)

• ScrapeCreators Instagram post_count: the payload's edge_owner_to_timeline_media.count is the page size (12) while has_next_page is true; the real total is not in the payload, so it is None unless the count exceeds the page or the page is the whole account.
• ScrapeCreators X reposts (RT @… or retweeted_status_result): likes, comments, saves are None (the vendor says 0), extras.is_repost.
• Bright Data X profile counts 0 beside posts: None (counts_withheld), tested.
• ScrapeCreators Pinterest "no public boards": a retryable SocialProviderError, not NotFoundError; the vendor answered empty for an account Bright Data reads, so it is never "the account does not exist" and never an empty profile.

Temporary: remove or disable before launch — test replay seam (REMOVE-BEFORE-LAUNCH)

transport.py register_social_replay(lookup, store) / ReplayHit: a host-registered pair asked before every ScrapeCreators GET and Bright Data scrape (a hit is returned instead of the request: no guard, no observer, nothing billed) and handed each real paid answer afterwards. Unregistered (every package test, every non-aidream host) it is a no-op; a seam error is logged and treated as a miss. aidream's gate (admins/fixtures, switch, expensive list) is aidream/services/runtime/admin_replay_cache.py. Delete the seam with the cache before go-live (rg REMOVE-BEFORE-LAUNCH).