tap-substack

A Singer tap for extracting data from Substack newsletters, built with the Meltano Singer SDK.

What's Working

Streams

Features

• Public + authenticated modes: Runs without auth for public data; add session_token to unlock dashboard/analytics streams
• Pagination: Offset-based pagination for list endpoints (posts, email_stats, subscribers)
• Rate limiting: Built-in 0.5s request throttle to avoid 429s
• Graceful auth errors: Authenticated streams log warnings and skip on 403/404/429 instead of crashing
• Parent-child streams: post_details and comments automatically iterate over posts from the posts stream
• Incremental replication: posts stream supports replication_key on post_date

Installation

pip install -e .

Configuration

Getting your session token

1. Log in to your Substack dashboard
2. Open browser DevTools > Application > Cookies
3. Copy the value of the substack.sid cookie

Usage

# Discovery mode
tap-substack --config sample_config.json --discover

# Sync public data only
tap-substack --config sample_config.json

# Pipe to a target
tap-substack --config config.json | target-jsonl

Development

# Set up
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

# Run against a test newsletter (public only)
tap-substack --config test_config.json