🎯 What This Solves

You hit a URL. It returns junk:

❌ "Please enable JavaScript"   ← x.com tweets, SPAs
❌ "Checking your browser..."   ← Cloudflare Turnstile
❌ HTTP 403 / 503               ← bot detection
❌ "internet-positif.info"      ← ISP DNS poison (🇮🇩)
❌ "Sign in to view"            ← login walls

unblock-web is a decision tree + verified scripts that pick the right tool for each block class. Drop it into any AI agent (Claude, Hermes, Cursor, Aider, your own) and stop guessing with raw curl/wget/playwright.

Status (May 2026): All 4 tiers verified working on Ubuntu 26.04 + WSL2.

✨ Features

🚀 Quick Start

Pick your favorite install method. All four work right now.

⚡ One-liner (zero-config)

curl -fsSL https://raw.githubusercontent.com/kevinnft/unblock-web/main/scripts/install.sh | bash

Picks a working Python (3.11–3.13), creates an isolated venv at ~/.unblock-web, installs Chromium via heal, and symlinks unblock-web into ~/.local/bin. Reversible: rm -rf ~/.unblock-web ~/.local/bin/unblock-web.

🐍 pip

pip install 'unblock-web[stealth]'
unblock-web heal              # one-time: auto-detects OS, installs Chromium
unblock-web verify            # 3-tier health check
unblock-web fetch https://x.com/elonmusk/status/123456789

🐳 Docker (zero-install)

docker run --rm ghcr.io/kevinnft/unblock-web:latest fetch https://example.com

# With TinyFish (Tier 2 geo-proxy)
docker run --rm \
  -e TINYFISH_API_KEY=$TINYFISH_API_KEY \
  ghcr.io/kevinnft/unblock-web:latest fetch https://blocked.com --proxy US

📦 From source

git clone https://github.com/kevinnft/unblock-web.git
cd unblock-web
pip install -e '.[stealth]'
unblock-web heal
unblock-web verify --verbose

🛠️ Library

from unblock_web import fetch

# Auto-pilot — picks the right tier per URL
page = fetch("https://x.com/seelffff/status/2055155782367187375")
print(page.text)
print(f"Used tier: {page.tier}")

# Force ISP/geo bypass
page = fetch("https://web3.okx.com", proxy_country="US")

# Force a specific tier
page = fetch("https://target.com", tier="T1", wait=8000)

🔌 In an AI agent

Hermes Agent example — drop the canary into session-start:

# ~/.hermes/config.yaml
hooks:
  on_session_start:
    - command: "unblock-web verify"
      timeout: 30

📖 Decision Tree

flowchart TD
    A[🌐 URL incoming] --> B{What kind of block?}

    B -->|Plain blog/docs/<br/>GitHub README| T0[⚡ Tier 0: scrapling.get<br/>fastest, no browser]
    B -->|JS-rendered SPA<br/>React/Next/Vue| T1[🛡️ Tier 1: stealthy_fetch<br/>+ network_idle + wait]
    B -->|Cloudflare Turnstile<br/>'Checking browser'| T1B[🛡️ Tier 1: stealthy_fetch<br/>+ solve_cloudflare=True]
    B -->|x.com tweet body| T1C[🛡️ Tier 1: stealthy_fetch<br/>captures DOM pre-modal]
    B -->|x.com replies/thread| T3[🪞 Tier 3: xcancel.com mirror<br/>via Tier 1 stealth]
    B -->|🇮🇩 ISP DNS block<br/>internet-positif| T2[🌍 Tier 2: TinyFish<br/>--proxy US]
    B -->|Geo-locked content| T2B[🌍 Tier 2: TinyFish<br/>--proxy XX]
    B -->|Login required<br/>DMs/private/paywall| T4[🔑 Tier 4: xurl + bearer<br/>or cookie injection]

    T0 --> R[✅ Markdown out]
    T1 --> R
    T1B --> R
    T1C --> R
    T3 --> R
    T2 --> R
    T2B --> R
    T4 --> R

    style T0 fill:#10b981,stroke:#059669,color:#fff
    style T1 fill:#f59e0b,stroke:#d97706,color:#fff
    style T1B fill:#f59e0b,stroke:#d97706,color:#fff
    style T1C fill:#f59e0b,stroke:#d97706,color:#fff
    style T2 fill:#06b6d4,stroke:#0891b2,color:#fff
    style T2B fill:#06b6d4,stroke:#0891b2,color:#fff
    style T3 fill:#a855f7,stroke:#9333ea,color:#fff
    style T4 fill:#ef4444,stroke:#dc2626,color:#fff
    style R fill:#22c55e,stroke:#16a34a,color:#fff

🛡️ The 4-Tier Stack

⚡ Tier 0: Plain HTTP

🛡️ Tier 1: Scrapling Stealth (PRIMARY)

StealthyFetcher.fetch(
    url,
    network_idle=True,        # wait for XHR settle
    solve_cloudflare=True,    # auto-handle Turnstile JS
    wait=5000,                # ms — let SPA hydrate
)

📚 Full param reference: docs/tier-1-scrapling.md

🌍 Tier 2: TinyFish (geo-proxy)

python3 scripts/tinyfish_fetch.py "https://blocked-site.com" --proxy US
python3 scripts/tinyfish_fetch.py --search "your query"  # bonus: free search API

📚 Setup + edge cases: docs/tier-2-tinyfish.md

🪞 Tier 3: Aggregator Mirrors

📚 Mirror rotation tips: docs/tier-3-mirrors.md

🔑 Tier 4: Authenticated APIs

📚 Step-by-step bearer setup: docs/tier-4-authenticated.md

🧪 Verified Targets

Stack was tested against these (May 2026) — every result is reproducible:

Reproduce these: see examples/ for runnable scripts.

🩺 Health Monitoring

Three layers, no cron required (built for laptops that sleep):

🚦 Session-start canary

Drop into any agent's session-start hook. Silent on healthy state, alert on regression:

# Hermes Agent example (~/.hermes/config.yaml)
hooks:
  on_session_start:
    - command: "/path/to/scripts/verify-stack.py"
      timeout: 30

🔧 Self-heal on Chromium loss

When stealthy_fetch errors with Executable doesn't exist (after a venv recreate), auto-run:

bash scripts/heal-chromium.sh

Idempotent. Safe to run anytime.

👀 On-demand audit

python3 scripts/verify-stack.py --verbose

🎨 Why "Anti-Blok"?

Because every "scraping tutorial" online stops at:

"Just install Playwright! Just use Selenium! Just pay for ScrapingBee!"

Then you hit the real world:

• 🇮🇩 ISP poisoning your DNS
• 🇨🇳 GFW dropping your packets
• ☁️ Cloudflare upgrading Turnstile every quarter
• 🐦 X.com adding login walls overnight
• 🐧 Ubuntu 26.04 breaking Playwright install

unblock-web is the field-tested decision tree from those battles. Free tools only. No API keys hoarded. Reproducible against listed targets.

📁 Repository Structure

unblock-web/
├── 📖 README.md              ← you are here
├── 📜 LICENSE                 ← MIT
├── 📚 docs/                   ← per-tier deep dives
│   ├── tier-1-scrapling.md
│   ├── tier-2-tinyfish.md
│   ├── tier-3-mirrors.md
│   ├── tier-4-authenticated.md
│   └── ubuntu-26-04-fix.md
├── 🛠️ scripts/                ← drop-in tools
│   ├── verify-stack.py        ← 3-tier canary
│   ├── heal-chromium.sh       ← Ubuntu 26.04 fix
│   └── tinyfish_fetch.py      ← Tier 2 wrapper
├── 🧪 examples/               ← reproducible cases
│   ├── x_com_tweet.py
│   ├── cloudflare_bypass.py
│   ├── indonesian_isp_bypass.py
│   └── xcancel_replies.py
├── ⚙️  .github/workflows/      ← CI canary
│   └── canary.yml
└── 🎨 assets/                 ← logo + diagrams

🤝 Contributing

Found a target the stack can't crack? Open an issue with:

1. ❓ The URL (or pattern)
2. 📋 What each tier returned (paste the failure)
3. 🤔 Hypothesis (login? CF v3? new anti-bot?)

Or send a PR to docs/known-targets.md when you find a workaround.

⚖️ Ethics & Legal

This stack is for reading publicly accessible content:

✅ Public tweets, blogs, docs, GitHub
✅ Content you're entitled to read in a browser
✅ APIs you have keys for

❌ Don't use it to:

• Scrape behind authentication you don't own
• Violate site Terms of Service
• Mass-extract copyrighted content
• Build credential-harvesting / phishing tools

Respect robots.txt. Respect rate limits. Be a good citizen of the open web.

🙏 Credits

Stack composed from:

• 🛡️ Scrapling — the unified scraping library
• 🥷 Patchright — anti-fingerprint Playwright fork
• 🐟 TinyFish — free fetch + search API for AI agents
• 🪞 xcancel.com — Twitter content mirror that survives
• 🐤 xurl — official X CLI