Socials Daily

Fetch recent posts from public social media accounts and generate a daily summary.

Setup

uv sync

Optional backends:

uv sync -E hikerapi -E xpoz -E youtube

Usage

1. Edit accounts.json — grouped by platform:

{
  "bluesky": {
    "accounts": [{"handle": "bsky.app"}]
  },
  "instagram": {
    "accounts": [{"handle": "natgeo"}, {"handle": "nasa"}]
  }
}

Each account can have a custom limit:

{
  "bluesky": {
    "accounts": [
      {"handle": "bsky.app"},
      {"handle": "atmos.bsky.social", "limit": 20}
    ]
  },
  "instagram": {
    "accounts": [
      {"handle": "natgeo"}
    ]
  }
}

Each platform can override the scraper backend (optional):

{
  "instagram": {
    "backend": "hikerapi",
    "accounts": [
      {"handle": "natgeo"}
    ]
  }
}

Backend resolution (highest to lowest priority):

1. CLI --backend flag (overrides everything)

2. Platform backend in accounts.json

3. Default mapping (instagram → instaloader, bluesky → bluesky, etc.)

4. Run the scraper:

# Default: Bluesky (free, no auth needed)
uv run python -m socials_daily scrape

# Instagram (free, rate-limited)
uv run python -m socials_daily scrape --backend instaloader

# HikerAPI (pay-per-request, ~$0.0006/request)
uv run python -m socials_daily scrape --backend hikerapi --api-key YOUR_KEY

# Xpoz (pre-indexed data, free tier available)
uv run python -m socials_daily scrape --backend xpoz --api-key YOUR_KEY

3. Check the output:

output/daily-summary-YYYY-MM-DD.md   # Markdown summary
output/daily-summary-YYYY-MM-DD.json # JSON for programmatic use

Add Accounts

# Bluesky (default)
uv run python -m socials_daily add bsky.app

# Instagram
uv run python -m socials_daily add natgeo --platform instagram

# With custom limit
uv run python -m socials_daily add atmos.bsky.social --platform bluesky --limit 20

# Set platform backend
uv run python -m socials_daily add natgeo --platform instagram --backend hikerapi

Scrapers

Bluesky (default)

Uses Bluesky's public AT Protocol API. No authentication required.

uv run python -m socials_daily scrape --backend bluesky

Instaloader

Free, open-source Instagram scraper. Rate-limited by Instagram.

HikerAPI

REST API with 100+ endpoints. No blocks, no rate limits.

export HIKERAPI_TOKEN=your-key
uv run python -m socials_daily scrape --backend hikerapi

Xpoz

Pre-indexed social data API. Supports Instagram, Twitter, TikTok, Reddit.

export XPOZ_API_KEY=your-key
uv run python -m socials_daily scrape --backend xpoz

YouTube

Fetches recent videos from YouTube channels. Requires optional yt-dlp dependency.

# Install YouTube support
uv sync -E youtube

# Add a channel (handle or name)
uv run python -m socials_daily add mkbhd --platform youtube

# Scrape
uv run python -m socials_daily scrape

Reddit

Fetches recent posts from public subreddits via Reddit's JSON API. No authentication required.

# Add a subreddit
uv run python -m socials_daily add programming --platform reddit

# Scrape
uv run python -m socials_daily scrape

RSS

Scrapes any RSS/Atom feed. The handle field holds the feed URL.

# Add an RSS feed
uv run python -m socials_daily add https://www.reddit.com/r/programming/.rss --platform rss

# Scrape
uv run python -m socials_daily scrape

Project Structure

accounts.json         # Accounts grouped by platform (with optional backend config)
src/socials_daily/    # Source code
├── __main__.py       # Entry point
└── scrapers/         # Scraper backends
    ├── base.py       # Abstract interface
    ├── bluesky.py
    ├── instaloader.py
    ├── reddit.py     # Reddit JSON API
    ├── rss.py        # RSS/Atom feeds
    ├── youtube.py    # YouTube channel videos
    ├── hikerapi.py
    └── xpoz.py
output/               # Generated daily summaries
pyproject.toml        # Project config (uv)

Architecture

See ARCHITECTURE.md for a detailed breakdown of the scraper layer, config resolution, deduplication system, and data flow.