signalforge-cli 0.15.0

CLI for extracting emails and sending customer.io events

   _____ _                   ________                    
  / ___/(_)___ _____  ____ _/ / ____/___  _________ ____ 
  \__ \/ / __ `/ __ \/ __ `/ / /_  / __ \/ ___/ __ `/ _ \
 ___/ / / /_/ / / / / /_/ / / __/ / /_/ / /  / /_/ /  __/
/____/_/\__, /_/ /_/\__,_/_/_/    \____/_/   \__, /\___/ 
       /____/                               /____/       

Mine developer signals. Enrich with emails. Fire into your CRM.

🟣 PyPI v0.7.2  |  🐍 Python 3.11+  |  📄 MIT  |  ⚡ Built on Backboard.io  |  📬 Customer.io  |  🏆 Devpost

SignalForge scrapes Devpost hackathons, GitHub forks, and RB2B visitor exports — enriches every lead with real emails — then fires them straight into Customer.io. One command. Hundreds of warm leads.

---

What's inside

Command	What it does
signalforge-devpost-search	Search Devpost by keyword → enrich with emails → export CSV
signalforge-participants	Scrape one hackathon's participants → CSV
signalforge-harvest	Walk the full hackathon listing → SQLite → delta Customer.io events
signalforge-devto	Walk dev.to challenges → scrape submitters → emails via GitHub → SQLite → emit events
signalforge-hn	Scrape HN Show posts → filter GitHub links → mine owner emails → SQLite → emit hacknews_posts events
signalforge-github-forks	Mine fork owners from any GitHub repo → emails → SQLite
signalforge-gh-search	Search GitHub repos by keyword → collect owner emails → SQLite
signalforge-gh-activity	Scrape PR authors + issue openers from tracked repos → email enrichment → Customer.io github_pull_request / github_issue events
signalforge-rb2b	Import RB2B visitor CSVs → SQLite → visited_site events
signalforge-auto	Full daily scrape: RB2B today + open hackathons + all tracked GitHub repos (no emit)
signalforge-emit-all	Flush every unsent event across all sources in one shot
signalforge-emit-batch	Emit up to --batch-size events per source bucket — cron-friendly
signalforge-auto-batch	One cron command: daily scrape + emit batch in a single run
signalforge-lookup	Search the DB by email, name, or username — show full lead context
signalforge-assistant	Interactive AI analyst REPL over your lead database
signalforge-campaigns	Sync email HTML files with Customer.io campaign actions via the App API

---

Install

pip install signalforge-cli

Or with uv (recommended for local dev):

uv sync

---

30-second quickstart

# 1. Copy env and fill in your keys
cp .env.example .env

# 2. Search Devpost and get a CSV of leads with emails
signalforge-devpost-search "ai agents" -o leads.csv

# 3. Scrape all open hackathons, enrich new participants, emit to Customer.io
signalforge-harvest --emit-events

---

How it works

  Devpost / GitHub / RB2B
         │
         ▼
  ┌─────────────────────┐
  │   fast scan / search │  (no enrichment yet — just IDs)
  └──────────┬──────────┘
             │
             ▼
  ┌─────────────────────┐
  │   SQLite upsert      │  detect NEW rows only (delta)
  └──────────┬──────────┘
             │
             ▼
  ┌─────────────────────┐
  │   email enrichment   │  GitHub API → profile walking → regex
  └──────────┬──────────┘
             │
             ▼
  ┌─────────────────────┐
  │   Customer.io emit   │  identify + track  (once per lead, ever)
  └─────────────────────┘

Delta logic: on re-runs, only new participants get the expensive enrichment. Already-emitted leads are never re-fired. Safe to run on a cron.

---

Environment

Copy .env.example → .env:

Variable	Required for	Notes
BACKBOARD_API_KEY	signalforge	Backboard account key
DEVPOST_ASSISTANT_ID	auto	Saved on first run, reused after
DEVPOST_SESSION	signalforge-participants, signalforge-harvest	_devpost cookie from browser DevTools
DEV_TO__DEVTO_FOREM_SESSION	signalforge-devto	_Devto_Forem_Session cookie from browser DevTools
DEV_TO_REMEMBER_USER_TOKEN	signalforge-devto	remember_user_token cookie (optional)
DEV_TO_CURRENT_USER	signalforge-devto	current_user cookie (optional)
GITHUB_TOKEN	optional	PAT for 5 000 req/hr vs 60. Zero scopes needed
CUSTOMERIO_SITE_ID	--emit-events	Customer.io Track API (event emission)
CUSTOMERIO_API_KEY	--emit-events	Customer.io Track API (event emission)
CUSTOMERIO_APP_API_KEY	signalforge-campaigns	Customer.io App API (campaign management)

---

Commands

signalforge-devpost-search — Devpost project search

Search Devpost by keyword, enrich each hit with the detail page + author email, export CSV.

signalforge-devpost-search "ai agents" --output results.csv
signalforge-devpost-search "climate tech" "developer tools" -o results.csv

---

signalforge-participants — single hackathon

Scrape one hackathon's full participant list.

# First time — hand over your session cookie
signalforge-participants "https://authorizedtoact.devpost.com/participants" \
  --jwt "<_devpost cookie value>" -o participants.csv

# Subsequent runs — reuses saved session
signalforge-participants "https://authorizedtoact.devpost.com/participants" -o out.csv

# Fast mode (skip email enrichment)
signalforge-participants "https://..." --no-email -o out.csv

# Enrich + emit to Customer.io in one shot
signalforge-participants "https://..." --emit-events -o out.csv

---

signalforge-harvest — full automated pipeline

Walks the Devpost hackathon listing, scrapes every participant, stores in SQLite, and emits Customer.io events for net-new leads.

# Standard run — open hackathons, 3 pages, enrich + emit
signalforge-harvest --emit-events

# Bulk first scrape without enrichment (fast)
signalforge-harvest --pages 5 --no-email

# Catch up: emit all unsent leads already in the DB (no scraping needed)
signalforge-harvest --emit-unsent

# Re-scan for new joiners, enrich + emit delta
signalforge-harvest --rescrape --emit-events

# Include ended hackathons too
signalforge-harvest --status open --status ended --pages 5

# Export everyone who has a LinkedIn URL but no email yet (CSV for manual outreach)
signalforge-harvest --export-linkedin -o linkedin_leads.csv

Flags

Flag	Default	Description
--pages N	3	Hackathon listing pages (9 hackathons each)
--hackathons N	0 (all)	Stop after the first N hackathons
--max-participants N	0 (unlimited)	Cap per hackathon
--jwt TOKEN	.env	Devpost _devpost session cookie
--db PATH	devpost_harvest.db	SQLite path
--status	open	open / ended / upcoming (repeatable)
--no-email	off	Skip enrichment entirely
--emit-events	off	Emit Customer.io events for delta participants
--emit-unsent	off	Just emit — no scraping
--rescrape	off	Re-scrape already-seen hackathons
--export-linkedin	off	Export CSV of all leads with LinkedIn but no email
--output / -o PATH	stdout	Output path for --export-linkedin

SQLite schema

• hackathons — url, title, org, state, dates, registrations, prize, themes, last_scraped_at
• participants — (hackathon_url, username) PK + enrichment fields + first_seen_at, last_seen_at, event_emitted_at

Customer.io events

Event name depends on how old the hackathon is:

Condition	Event name
Hackathon is open, or closed within the last 30 days	devpost_hackathon
Hackathon closed more than 30 days ago	closed_hackathon

Email = Customer.io user ID. Payload: hackathon_url, hackathon_title, username, name, specialty, profile_url, github_url, linkedin_url.

Email templates: emails/devpost-hackathon/ (variants a–l) for devpost_hackathon, emails/closed-hackathon/ (variants a–f, MLH free-tier campaign) for closed_hackathon. All use {{customer.first_name}} and {{event.*}} Liquid variables. Push changes to cx.io with signalforge-campaigns update-all.

---

signalforge-devto — dev.to challenge submitters

Walk all active and previous dev.to challenges, scrape every submission article, extract author GitHub usernames from the dev.to API, look up real emails via GitHub, and store everything in SQLite.

How it works:

1. Scrapes https://dev.to/challenges for active + previous challenges
2. Fetches each challenge detail page to find the submission tag (e.g. #418challenge)
3. Paginates GET https://dev.to/api/articles?tag={tag} — no auth needed for articles
4. Extracts user.github_username directly from the API response
5. Looks up email via GitHub API (3 strategies: profile → commits → push events)
6. Deduplicates by (challenge_tag, dev.to username) — only new submitters trigger events

# First-time scrape: active + previous challenges, enrich emails
signalforge-devto

# Active challenges only
signalforge-devto --state active

# Skip email enrichment (fast scan)
signalforge-devto --no-email

# Emit Customer.io events for new submitters
signalforge-devto --emit-events

# Emit all unsent events in the DB (no scraping)
signalforge-devto --emit-unsent

# Re-scrape already-seen challenges (picks up new submissions)
signalforge-devto --rescrape --emit-events

# Cap submissions per challenge
signalforge-devto --max-submissions 50

Required env — copy from browser DevTools → Application → Cookies → dev.to:

DEV_TO__DEVTO_FOREM_SESSION=<value of _Devto_Forem_Session>
DEV_TO_REMEMBER_USER_TOKEN=<value of remember_user_token>   # optional
DEV_TO_CURRENT_USER=<value of current_user>                  # optional

Flags

Flag	Default	Description
--state	active+previous	active / previous / upcoming (repeatable)
--max-submissions N	0 (unlimited)	Cap submissions per challenge
--no-email	off	Skip email enrichment entirely
--emit-events	off	Emit devto_challenge Customer.io events for new submitters
--emit-unsent	off	Just emit — no scraping
--rescrape	off	Re-scrape already-scraped challenges
--db PATH	devpost_harvest.db	SQLite path

SQLite storage — reuses the shared participants table:

• hackathon_url = devto:challenge:{tag} (e.g. devto:challenge:418challenge)
• hackathon_title = challenge title
• username = dev.to username
• profile_url = https://dev.to/{username}
• github_url = extracted from dev.to API user object
• specialty = article URL

Customer.io event: devto_challenge Payload: challenge_url, challenge_title, username, name, profile_url, github_url, article_url.

---

signalforge-hn — Hacker News Show HN

Scrapes https://news.ycombinator.com/show, filters to posts that link to GitHub, extracts the repo owner username, looks up real emails via the GitHub API, and stores everything in SQLite under the hacknews_posts campaign.

How it works:

1. Fetches HN Show pages (30 posts/page)
2. Keeps only posts whose URL points to github.com
3. Extracts the GitHub repo owner (first path segment of the URL)
4. De-duplicates by owner — each GitHub user appears at most once
5. Looks up email via GitHub API (3 strategies: profile → commits → push events)
6. Stores as HackathonParticipant with hackathon_url=hn:show
7. Optionally emits hacknews_posts Customer.io events

# Scrape page 1 of HN Show and enrich emails
signalforge-hn

# Scrape 3 pages (~90 posts)
signalforge-hn --pages 3

# Fast mode — no email enrichment
signalforge-hn --pages 2 --no-email

# Scrape + emit Customer.io events in one shot
signalforge-hn --emit-events

# Cap emit to 20 contacts (for testing)
signalforge-hn --emit-events --emit-limit 20

# Skip scraping — just emit all queued unsent events
signalforge-hn --emit-unsent

# Re-enrich already-known owners
signalforge-hn --force-email

Flags

Flag	Default	Description
--pages N	1	HN Show pages to scrape (~30 posts each)
--db PATH	devpost_harvest.db	SQLite path
--no-email	off	Skip enrichment
--force-email	off	Re-enrich owners already in the DB
--emit-events	off	Emit hacknews_posts Customer.io events
--emit-limit N	0 (all)	Cap emit to N contacts
--emit-unsent	off	Emit only — no scraping

Customer.io event: hacknews_posts Payload: post_title, github_url, username, profile_url.

---

signalforge-github-forks — GitHub fork mining

Pull every fork owner from a repo, enrich with public emails, store in the same SQLite DB.

# Built-in presets
signalforge-github-forks --preset mem0 --emit-events
signalforge-github-forks --preset supermemory --no-email

# Any repo
signalforge-github-forks --repo owner/repo --limit 1000 --mode first_n

Flag	Default	Description
--preset	—	mem0 or supermemory shorthand
--repo OWNER/REPO	—	Any public GitHub repo
--limit N	2000	Max forks to process
--mode	preset-dependent	top_by_pushed or first_n
--no-email	off	Skip email lookup
--emit-events	off	Emit Customer.io events
--force-email	off	Re-enrich all forks, not just new ones

---

signalforge-auto — full daily scrape

Runs all three scrapers in sequence, then exits without emitting events. Use this as your daily cron job; fire --emit-unsent on each source afterwards.

What it runs:

1. signalforge-rb2b --fetch-date TODAY — pulls today's RB2B visitor export
2. signalforge-harvest --status open --pages 100 — walks all open Devpost hackathons
3. signalforge-github-forks --repo OWNER/REPO --limit 5000 — for every repo already tracked in the DB

# Standard daily run
signalforge-auto

# Custom date / page depth
signalforge-auto --fetch-date 2026-03-31 --pages 50 --fork-limit 2000

# Skip email enrichment (much faster, enrich later with --force-email)
signalforge-auto --no-email

# Then flush the queue when ready
signalforge-harvest --emit-unsent
signalforge-github-forks --emit-unsent
signalforge-rb2b --emit-unsent

Flag	Default	Description
--db PATH	devpost_harvest.db	SQLite path
--pages N	100	Devpost listing pages
--fork-limit N	5000	Max forks per GitHub repo
--fetch-date YYYY-MM-DD	today	RB2B export date
--no-email	off	Skip email enrichment
--jwt TOKEN	.env	Devpost session cookie

---

signalforge-rb2b — RB2B visitor import

Load RB2B daily export CSVs and fire visited_site events for identified visitors.

# Import and emit new identified visitors
signalforge-rb2b daily_2026-03-*.csv --emit-events

# Just drain the unsent queue
signalforge-rb2b --emit-unsent

---

signalforge-gh-search — GitHub repo search

Search GitHub repos by keyword, collect owner emails via the GitHub API, and store results in the harvest DB (hackathon_url = github:search:<query-slug>).

# Search and enrich owners
signalforge-gh-search "ai memory" --max 200 --emit-events

# Top results by stars (default), forks, or recency
signalforge-gh-search "langchain rag" --sort forks --max 500

# Skip enrichment now, drain later
signalforge-gh-search "vector database" --no-email
signalforge-gh-search --emit-unsent

Flag	Default	Description
query	—	GitHub search query (e.g. "AI memory")
--max N	100	Max repos to retrieve (GitHub caps at 1000)
--sort	stars	stars / forks / updated
--db PATH	devpost_harvest.db	SQLite path
--no-email	off	Skip email enrichment
--force-email	off	Re-enrich owners already in the DB
--emit-events	off	Emit github_search events to Customer.io
--emit-limit N	0 (all)	Cap --emit-events to N owners
--emit-unsent	off	Skip search — flush unsent queue only

---

signalforge-gh-activity — PR authors + issue openers from tracked repos

Scrapes pull-request authors and/or issue openers from GitHub repos already tracked in the DB (via signalforge-github-forks and signalforge-gh-search runs), enriches each user with a real email via the GitHub profile API, and stores results as two new campaign types:

• github:pull_requests:owner/repo — PR authors → Customer.io event github_pull_request
• github:issue_list:owner/repo — issue openers → Customer.io event github_issue

# Scrape PRs + issues from all tracked repos (forks + search), enrich emails
signalforge-gh-activity

# PRs only, from fork-tracked repos only
signalforge-gh-activity --type prs --source forks

# Issues only, from a specific repo
signalforge-gh-activity --type issues --repo mem0ai/mem0

# Fast scan (no email enrichment)
signalforge-gh-activity --no-email

# Scrape + emit Customer.io events in one shot
signalforge-gh-activity --emit-events

# Flush unsent events without re-scraping
signalforge-gh-activity --emit-unsent

# Limit to the 200 most recent items per repo (default), emit a test batch of 5
signalforge-gh-activity --max 200 --emit-events --emit-limit 5

Flags

Flag	Default	Description
--type	both	prs / issues / both — what to scrape
--source	both	forks / search / both — which tracked repos to use
--repo OWNER/REPO	—	Scrape a specific repo (can be combined with --source)
--max N	200	Max PRs/issues to fetch per repo
--state	all	all / open / closed — GitHub filter
--db PATH	devpost_harvest.db	SQLite path
--no-email	off	Skip email enrichment — just store users
--force-email	off	Re-enrich users already in the DB
--emit-events	off	Emit github_pull_request / github_issue events to Customer.io
--emit-limit N	0 (all)	Cap --emit-events to N users per repo
--emit-unsent	off	Skip scraping — flush the unsent queue only

SQLite storage — reuses the shared participants table:

Field	PR rows	Issue rows
hackathon_url	github:pull_requests:owner/repo	github:issue_list:owner/repo
hackathon_title	GitHub PRs: owner/repo	GitHub issues: owner/repo
username	GitHub login	GitHub login
github_url	profile URL	profile URL
specialty	PR URL	Issue URL
email	from GitHub profile API	from GitHub profile API

Customer.io events

Source	Event name	Key payload fields
PR authors	github_pull_request	repo_name, repo_url, pr_url, username, github_url + profile tier fields
Issue openers	github_issue	repo_name, repo_url, issue_url, username, github_url + profile tier fields

---

signalforge-emit-all — flush all unsent events

Drain the unsent queue for every source in one shot: Devpost hackathons, GitHub fork owners, GitHub search owners, GitHub PR authors, GitHub issue openers, and RB2B visitors.

signalforge-emit-all
signalforge-emit-all --db my_harvest.db

Flag	Default	Description
--db PATH	devpost_harvest.db	SQLite path

---

signalforge-auto-batch — daily scrape + emit in one command

The single cron entry you actually need. Runs the full daily scrape (signalforge-auto) and then immediately emits one batch from every source bucket (signalforge-emit-batch).

# Add to crontab — runs at 6 AM daily
0 6 * * * /path/to/venv/bin/signalforge-auto-batch >> /var/log/signalforge.log 2>&1

# Manual run with defaults
signalforge-auto-batch

# Smaller emit batch (useful during warm-up while the queue is large)
signalforge-auto-batch --batch-size 500

# Skip email enrichment for a faster run
signalforge-auto-batch --no-email --batch-size 1000

Flag	Default	Description
--batch-size N	2000	Max events to emit per source bucket after scrape
--pages N	100	Devpost listing pages to fetch
--fork-limit N	5000	Max forks per GitHub repo
--fetch-date YYYY-MM-DD	today	RB2B export date
--no-email	off	Skip email enrichment
--jwt TOKEN	.env	Devpost session cookie
--db PATH	devpost_harvest.db	SQLite path

---

signalforge-emit-batch — batched emit (cron-friendly)

Emit up to --batch-size events from each of four source buckets in a single run. Unlike signalforge-emit-all (which drains the entire queue), this lets you pace delivery — run it on a cron until the queue is empty.

Bucket	Event name	Source
Devpost open / recently-closed (≤30 days)	devpost_hackathon	harvest DB
Devpost old-closed (>30 days)	closed_hackathon	harvest DB
GitHub fork owners	github_fork	harvest DB
RB2B identified visitors	visited_site	harvest DB

# Emit up to 2000 per bucket (default)
signalforge-emit-batch

# Smaller batches — good for warming up or rate-limiting
signalforge-emit-batch --batch-size 500

# Custom DB
signalforge-emit-batch --batch-size 1000 --db my_harvest.db

Flag	Default	Description
--batch-size N	2000	Max events to emit per source bucket
--db PATH	devpost_harvest.db	SQLite path

---

signalforge-lookup — contact lookup

Search the SQLite DB by email address, name, or username and print the full lead record with hackathon context.

signalforge-lookup [e-mail]
signalforge-lookup "Alice Smith"
signalforge-lookup alicedev
signalforge-lookup alice --db my_harvest.db

Flag	Default	Description
query	—	Email, name, or username to search
--db PATH	devpost_harvest.db	SQLite path

---

signalforge-assistant — AI analyst REPL

Interactive natural-language interface to your lead database powered by a Backboard AI assistant. Ask questions, query leads, and get summaries without writing SQL.

signalforge-assistant
signalforge-assistant --db my_harvest.db

> How many participants have emails from the last 30 days?
> Show me leads from AI-themed hackathons with a GitHub URL
> Which hackathons have the most unemitted participants?

Flag	Default	Description
--db PATH	devpost_harvest.db	SQLite path

Requires BACKBOARD_API_KEY. Model defaults to gpt-4o-mini; override with BACKBOARD_MODEL and BACKBOARD_LLM_PROVIDER.

---

signalforge-campaigns — Customer.io campaign management

Sync HTML email files with Customer.io campaign actions via the App API. Eliminates copy-pasting: edit locally, push with one command.

Email templates live in emails/ organised by campaign:

emails/
├── manifest.json                  ← global action ↔ file registry
├── campaigns/                     ← per-campaign fetched manifests
│   └── {campaign_id}.json
├── devpost-hackathon/             ← variants for the devpost_hackathon event
│   ├── variant-a.html … variant-l.html
├── closed-hackathon/              ← variants for the closed_hackathon event (MLH free tier)
│   ├── variant-a.html … variant-f.html
└── github-forkers/                ← variants for the github_fork event
    └── variant-a.html … variant-h.html

Every HTML file has a <!-- Subject: ... --> comment at line 1 — that becomes the email subject on push. Liquid variables {{customer.first_name}} and {{event.*}} are used throughout.

Typical workflow:

# 1. See all campaigns
signalforge-campaigns list-campaigns

# 2. Pull a campaign + all its actions into emails/campaigns/{id}.json
signalforge-campaigns get-campaign --campaign-id 9

# 3. Visualise the action graph as a Mermaid flowchart
signalforge-campaigns show-campaign --campaign-id 9

# 4. Auto-pair all email actions to a folder of HTML files (checks counts match)
signalforge-campaigns get-actions --campaign-id 9 --folder emails/closed-hackathon

# 5. Push all files for a campaign at once
signalforge-campaigns update-all --campaign-id 9

# One-off: register or push a single action
signalforge-campaigns get    --campaign-id 9 --action-id 353 --file emails/closed-hackathon/variant-c.html
signalforge-campaigns update --file emails/closed-hackathon/variant-c.html

Subcommands

Subcommand	Description
list-campaigns	List all campaigns — id, state, name
get-campaign	Fetch a campaign + all its actions → emails/campaigns/{id}.json
show-campaign	Print a Mermaid flowchart of the campaign's action graph
get-actions	Fetch all email actions, pair with a local folder, upsert manifest.json
update-all	Push every manifest-linked HTML file for a campaign to cx.io
get	Fetch one action and upsert it into manifest.json
update	Push one local HTML file's subject + body to cx.io

get-actions count validation

Before writing anything, get-actions compares the number of email-type actions in the campaign (A/B test container nodes are automatically excluded) against the number of HTML files in the folder. If they don't match, it prints both lists and exits — nothing is written.

# Skip the confirmation prompt for scripting / CI
signalforge-campaigns get-actions --campaign-id 9 --folder emails/closed-hackathon --yes

manifest.json schema

[
  {
    "file":            "emails/closed-hackathon/variant-c.html",
    "campaign_id":     "9",
    "action_id":       "353",
    "name":            "Email 1",
    "subject":         "Late to your inbox, but this is worth it",
    "last_fetched_at": "2026-04-02T18:51:27+00:00",
    "last_pushed_at":  "2026-04-02T18:51:51+00:00"
  }
]

Commit manifest.json alongside the HTML files so the team can always see which version is live in cx.io and when it was last pushed.

---

Requirements

• Python 3.11+
• uv (for local dev)
• Backboard API key (for signalforge-devpost-search keyword search only)

---

Development

uv run python -m devpost_scraper.cli "ai agents" --output out.csv