Apophenia

"The human tendency to perceive meaningful patterns within random data."

Apophenia is a data pipeline that finds absurd, statistically significant correlations between completely unrelated datasets — FRED economic indicators, Google Trends searches, OSHA workplace injuries, USPTO patent filings, and ESPN sports viewership. It scores each result on absurdity and generates clickbait headlines.

---

Features

• Multi-source data ingestion — FRED, Google Trends, OSHA, USPTO patents, ESPN, Yahoo Finance
• Rigorous statistics — Pearson / Spearman correlation, lagged cross-correlation (up to 30 days), rolling correlation, cointegration (Engle-Granger)
• Absurdity scoring — Weighted combination of domain distance, statistical strength, and directional inconsistency
• Clickbait headline generator — "Economists baffled: SP500 skyrocketed 80% while Bigfoot searches crashed 65%"
• CLI + REST API — Use from the terminal or via HTTP
• Disk-cached — No redundant API calls
• LLM support — Optional OpenAI integration for AI-powered absurdity scoring

---

Quick Start

# Install
pip install apophenia-trend

# Or from source
git clone https://github.com/sectersion/apophenia.git
cd apophenia
pip install -e .

# Set up your API keys (optional — most sources work without them)
cp .env.example .env
# Edit .env to add FRED_API_KEY and/or OPENAI_API_KEY

CLI

# Find the most absurd correlations
apophenia --start 2020-01-01 --end 2024-01-01 --min-absurdity 0.6

# Output as JSON
apophenia --start 2022-01-01 --end 2023-01-01 --json

API

# Start the server
uvicorn apophenia.api:app --reload

# Or use the CLI shortcut
python -c "from apophenia.api import serve; serve()"

curl "http://localhost:8000/correlate?start=2020-01-01&end=2024-01-01&min_absurdity=0.7"

---

Data Sources

Source	Category	Requires Key	Description
FRED	Economics	Key (free)	GDP, unemployment, fed funds, CPI, yield curve, VIX, housing starts, industrial production...
Google Trends	Culture	No	Search volume for 20+ queries ("bitcoin", "quiet quitting", "aliens", "wordle"... )
OSHA	Health	No	Workplace injury / fatality enforcement data (falls, electrocutions, amputations...)
USPTO Patents	Innovation	No	Patent filings for 13 categories (AI, blockchain, gene editing, spacecraft...)
ESPN	Sports	No	Scraped viewership / interest across 10 leagues (NFL, NBA, MLB, UFC, golf...)
Yahoo Finance	Finance	No	30+ tickers (SPY, QQQ, GLD, BTC-USD, TLT, sector ETFs, VIX...)

---

Architecture

src/apophenia/
├── __init__.py          # Cache directory setup
├── config.py            # Pydantic settings (env vars)
├── cli.py               # Rich-formatted terminal UI
├── api.py               # FastAPI REST server
├── core/
│   ├── engine.py        # Orchestration pipeline
│   ├── correlation.py   # Statistical computation
│   ├── absurdity.py     # Absurdity scoring
│   └── headlines.py     # Clickbait headline generator
└── sources/
    ├── base.py          # Abstract source + caching
    ├── fred.py          # Federal Reserve data
    ├── trends.py        # Google Trends
    ├── osha.py          # OSHA enforcement
    ├── patents.py       # USPTO filings
    ├── espn.py          # ESPN sports data
    └── yfinance_source.py  # Yahoo Finance market data

Pipeline

Sources ─► Fetch ─► Align ─► Correlate ─► Score ─► Headline ─► Results
  │          │          │           │          │           │
  │    diskcache    dropna     Pearson    domain     random
  │    + rate      + union    Spearman   distance   templates
  │    limits       index      lag(1..30) stat.strength
  │                            rolling   direction
  │                            coint

---

Absurdity Score

Each correlation gets a score [0, 1] based on three components:

Component	Weight	Description
Category Distance	30%	How unrelated the domains are (e.g., sports × health = 0.5, sports × economics = 0.8)
Statistical Strength	40%	Magnitude of r, significance of p, with bonus for extreme values
Direction Absurdity	30%	Whether contemporaneous and lagged correlations point in opposite directions

The higher the score, the more absurd (and interesting) the result.

---

Example Output

┌─────────────────────────────────────────────────────────────────────┐
│               Apophenia Report — Top 5 Correlations                │
├───┬──────────┬────────────────────────────────────────┬──────┬─────┤
│ # │ Absurdity │ Headline                              │ r    │ p   │
├───┼──────────┼────────────────────────────────────────┼──────┼─────┤
│ 1 │     0.89 │ SP500 skyrocketed 83% — Meanwhile,     │ 0.92 │ 2e-4│
│   │          │ Bigfoot searches plummeted 72%         │      │     │
│ 2 │     0.84 │ Can Electrocutions predict NFT hype?   │ 0.87 │ 5e-3│
│   │          │ Crashed 61% foreshadows Surging 78%    │      │     │
│ 3 │     0.81 │ Coincidence? Federal Funds Rate soared │ 0.85 │ 1e-3│
│   │          │ and Divorce Lawyer searches tanked     │      │     │
│ 4 │     0.78 │ As 3D Printing jumped 67%, NBA         │ 0.81 │ 8e-3│
│   │          │ viewership collapsed 54%               │      │     │
│ 5 │     0.72 │ Economists baffled: Unemployment       │-0.79 │ 3e-2│
│   │          │ nosedived 62% correlates with Keto     │      │     │
│   │          │ Diet searches exploding 91%            │      │     │
└───┴──────────┴────────────────────────────────────────┴──────┴─────┘

---

Development

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Lint
ruff check src/

# Type check
mypy src/

---

Configuration

All settings via .env or environment variables:

Variable	Required	Description
FRED_API_KEY	Optional	FRED API key (free signup)
OPENAI_API_KEY	Optional	OpenAI key for LLM-based absurdity scoring

---

Why "Apophenia"?

Apophenia is the tendency to perceive meaningful connections between unrelated things. The term was coined by neurologist Klaus Conrad in 1958. This project leans hard into that tendency — with statistics.

---

License

MIT