---

What is IntelliScrape?

IntelliScrape is a Python web scraping library that scrapes 98% of websites out of the box. It uses a 4-tier engine system that automatically escalates from fast HTTP requests to full browser automation — you get the cheapest, fastest method that works, and heavier weapons only when needed.

No more switching between requests, playwright, and selenium. No more debugging why your scraper got blocked. Just scrape(url) and you're done.

Key capabilities:

• 4-tier engine escalation (static → Playwright → Camoufox → nodriver)
• TLS fingerprint impersonation (JA3/JA4 bypass)
• Browser fingerprint randomization
• Human-like behavioral simulation
• CAPTCHA detection, automated solving, and manual solving
• Anti-bot vendor detection (Cloudflare, Akamai, DataDome, PerimeterX)
• Intelligent site analysis and auto-configuration
• Proxy rotation with free proxy finder
• Export to JSON, CSV, Excel, SQLite, Text, Markdown
• Async support for concurrent scraping
• Link checking (status verification, categorization, broken link detection)
• Website mirroring (HTTrack-ported, WARC/ZIP export, offline browsing)

---

Installation

pip install intelliscrape

Optional Extras

Extra	Command	What it adds
stealth	pip install intelliscrape[stealth]	nodriver engine (anti-WebDriver detection)
camoufox	pip install intelliscrape[camoufox]	Camoufox engine (Firefox-based, C++ patches)
captcha	pip install intelliscrape[captcha]	CapSolver integration (reCAPTCHA, hCaptcha, Turnstile)
async	pip install intelliscrape[async]	Async/concurrent scraping
all	pip install intelliscrape[all]	Everything above
dev	pip install intelliscrape[dev]	pytest, ruff

---

Quick Start

One-liner (Python)

from intelliscrape import scrape

text = scrape("https://example.com")
print(text[:500])

CLI

intelliscrape https://example.com

Full-featured class

from intelliscrape import IntelliScrape

scraper = IntelliScrape()
result = scraper.scrape("https://example.com")
print(result)

With proxy

scraper = IntelliScrape(proxy="user:pass@proxy:8080")
result = scraper.scrape("https://protected-site.com")

Get structured data

scraper = IntelliScrape()
data = scraper.get_structured("https://github.com")

print(data.title)          # Page title
print(data.description)    # Meta description
print(data.og_data)        # OpenGraph tags
print(data.json_ld)        # JSON-LD structured data

Crawl entire website

from intelliscrape import crawl

result = crawl("https://docs.python.org", max_pages=100)
print(f"Scraped {result.total_pages} pages")

for page in result.pages:
    print(f"  {page.url}: {len(page.content)} chars")

---

CLI Reference

intelliscrape [URL] [OPTIONS]

Output

Flag	Description
-o, --output FILE	Save output to file
--json	Structured JSON (title, description, meta tags)
--raw	Raw HTML instead of extracted text

Intelligent Mode

Flag	Description
--analyze	Analyze site and show recommendations
--no-intelligent	Disable intelligent auto-detection

Engine

Flag	Description
--force-browser	Force browser engine for JS-heavy sites
--manual-captcha	Open visible browser for manual CAPTCHA solving

Proxy

Flag	Description
--use-free-proxies	Use free proxies automatically
--no-free-proxies	Disable free proxy finder
--find-proxies	Find and test free proxies (no scraping)
--brightdata-key KEY	Bright Data API key
--scraperapi-key KEY	ScraperAPI key
--oxylabs-key KEY	Oxylabs API key
--smartproxy-key KEY	Smartproxy API key

Authentication

Flag	Description
--login	Login before scraping
--username USER	Username/email
--password PASS	Password
--login-url URL	Explicit login URL

Cookies

Flag	Description
--save-cookies FILE	Save cookies to JSON
--load-cookies FILE	Load cookies from JSON

Request Modification

Flag	Description
--block PATTERNS	Block URLs (comma-separated)
--header "Key: Value"	Add custom header (repeatable)

Pagination & Search

Flag	Description
--paginate	Auto-follow pagination
--max-pages N	Max pages (default: 50)
--search QUERY	Submit search query

Crawl

Flag	Description
--crawl	Crawl entire website

Link Checking

Flag	Description
--check-links	Check all links on the page and report status

Downloads

Flag	Description
--download	Download linked files
--download-images	Download all images
--download-dir DIR	Download directory (default: downloads)

Mirror (HTTrack-style)

Flag	Description
--mirror	Mirror entire website for offline browsing
--mirror-depth N	Max recursion depth (default: 5)
--mirror-output DIR	Output directory (default: ./mirror)
--mirror-zip FILE	Also create ZIP archive
--mirror-warc FILE	Also create WARC archive
--mirror-delay SEC	Delay between requests (default: 0.5)
--mirror-exclude PAT	Exclude URL patterns (repeatable)
--mirror-include PAT	Include URL patterns (repeatable)
--mirror-engine ENG	Engine: static, playwright, camoufox, nodriver, auto
--mirror-proxy URL	Proxy for mirroring
--mirror-update	Resume/update existing mirror
--no-robots	Ignore robots.txt

Export

Flag	Description
--export FORMAT	json, csv, excel, sqlite, text, markdown

Examples

# Basic
intelliscrape https://example.com -o output.txt

# Structured data
intelliscrape https://example.com --json

# Analyze protection
intelliscrape https://amazon.com --analyze

# Free proxies
intelliscrape https://amazon.com --use-free-proxies

# Login
intelliscrape https://site.com --login --username user --password pass

# Pagination
intelliscrape https://example.com/products --paginate --max-pages 10

# Crawl
intelliscrape https://docs.python.org --crawl --max-pages 50

# Check links
intelliscrape https://example.com --check-links
intelliscrape https://example.com --check-links --export json -o report.json

# Export
intelliscrape https://example.com --export csv -o data.csv

# Manual CAPTCHA
intelliscrape https://protected-site.com --manual-captcha

# Force browser
intelliscrape https://react-app.com --force-browser

# Mirror entire site
intelliscrape https://example.com --mirror --mirror-depth 3 --mirror-output ./backup

# Mirror + ZIP
intelliscrape https://example.com --mirror --mirror-zip site.zip

# Mirror + WARC
intelliscrape https://example.com --mirror --mirror-warc archive.warc.gz

# Mirror with proxy
intelliscrape https://example.com --mirror --mirror-proxy socks5://proxy:1080

# Mirror excluded patterns
intelliscrape https://example.com --mirror --mirror-exclude "*.pdf" --mirror-exclude "/admin/*"

---

Library API Reference

scrape() — Quick One-liner

from intelliscrape import scrape

text = scrape(url, **kwargs)

Parameter	Type	Default	Description
url	str	required	Target URL
engine	str	None	Force engine: static, playwright_stealth, camoufox, nodriver
extract	bool	True	Extract text from HTML
clean	bool	True	Clean extracted text
return_raw	bool	False	Return raw HTML
return_structured	bool	False	Return StructuredData
handle_consent	bool	True	Handle cookie consent banners
force_browser	bool	False	Force browser engine

---

IntelliScrape — Main Class

Constructor

from intelliscrape import IntelliScrape

scraper = IntelliScrape(**kwargs)

Parameter	Type	Default	Description
proxy	str, ProxyConfig, list	None	Single proxy or list
proxies	list of str	None	Proxy strings
brightdata_key	str	None	Bright Data API key
scraperapi_key	str	None	ScraperAPI key
oxylabs_key	str	None	Oxylabs API key
smartproxy_key	str	None	Smartproxy API key
prefer_residential	bool	True	Prefer residential proxies
use_free_proxies	bool	True	Auto-find free proxies
api_key	str	None	CAPTCHA solving API key
captcha_provider	str	None	2captcha or capsolver
headless	bool	True	Headless browser mode
simulate_behavior	bool	True	Human-like behavior simulation
manual_captcha	bool	False	Manual CAPTCHA solving mode
tls_profile	str	chrome131	TLS fingerprint profile
session_profile	str	None	Persistent session name
max_retries	int	3	Max retry attempts
min_delay	float	0.5	Min delay between requests
max_delay	float	3.0	Max delay between requests
requests_per_minute	int	None	Rate limit
intelligent	bool	True	Enable intelligent mode
log_level	str	WARNING	Logging level

Methods

scrape(url, **kwargs)

Scrape a URL and return text content.

result = scraper.scrape(
    url="https://example.com",
    engine=None,
    extract=True,
    clean=True,
    return_raw=False,
    return_structured=False,
    handle_consent=True,
    force_browser=False,
    intelligent=None,
)

get_structured(url, **kwargs)

Get structured data (title, description, meta tags, JSON-LD).

data = scraper.get_structured("https://github.com")
print(data.title)
print(data.description)
print(data.og_data)
print(data.json_ld)

analyze(url)

Analyze a site and return recommendations.

analysis = scraper.analyze("https://amazon.com")
print(analysis.site_type)           # "ecommerce"
print(analysis.protection_level)    # "high"
print(analysis.recommended_engine)  # "playwright_stealth"
print(analysis.recommended_delay)   # 3.0

scrape_many(urls, **kwargs)

Scrape multiple URLs with rate limiting.

results = scraper.scrape_many([
    "https://example.com/page1",
    "https://example.com/page2",
])
# Returns: [{"url": ..., "content": ..., "success": ..., "error": ...}, ...]

check_captcha(url)

Check if a URL has a CAPTCHA.

captcha = scraper.check_captcha("https://site.com")
if captcha:
    print(captcha.captcha_type)  # CaptchaType.RECAPTCHA_V2
    print(captcha.site_key)

check_antibot(url)

Check anti-bot protection on a URL.

info = scraper.check_antibot("https://site.com")
if info:
    print(info.vendor)       # AntiBotVendor.CLOUDFLARE
    print(info.confidence)   # 0.95

check_links(url, **kwargs)

Check all links on a page and return a detailed report with status codes, categorization, and summary statistics.

report = scraper.check_links("https://example.com", ignore_external=True)

print(f"Total links: {report.summary.total}")
print(f"OK: {report.summary.ok}, Broken: {report.summary.broken}")
print(f"Success rate: {report.summary.success_rate:.1f}%")
print(f"Internal: {report.summary.internal}, External: {report.summary.external}")
print(f"By type: {report.summary.by_type}")

# Per-link details
for link in report.links:
    print(f"  {link.url} -> {link.status_code} ({link.status.value})")

Parameter	Type	Default	Description
url	str	required	Page URL to check
timeout	int/float	5	Per-request timeout in seconds
ignore_external	bool	False	Skip external links
max_workers	int	10	Concurrent threads for checking
allowed_statuses	sequence	200-399	HTTP codes considered "OK"

Returns LinkCheckReport with:

• report.links — list of SingleLinkResult (url, status_code, status, link_type, is_external)
• report.summary — LinkCheckSummary with aggregate stats
• report.summary.by_type — breakdown by link type (page, image, video, etc.)

Standalone function:

from intelliscrape import check_links

report = check_links("https://example.com")
print(f"Broken: {report.summary.broken}")

find_free_proxies(test=True)

Find and test free proxies.

proxies = scraper.find_free_proxies(test=True)
for p in proxies:
    print(f"{p['url']} - speed: {p['speed']:.2f}s")

get_proxy_status()

Get proxy manager status.

status = scraper.get_proxy_status()
print(status['user_proxies'])
print(status['healthy_proxies'])

---

crawl() — Website Crawler

from intelliscrape import crawl

result = crawl(
    url="https://docs.python.org",
    max_pages=50,
    delay=0.5,
    on_page=None,  # Callback: on_page(done, failed)
)

Parameter	Type	Default	Description
url	str	required	Starting URL
max_pages	int	50	Maximum pages to crawl
delay	float	0.5	Delay between requests
on_page	callable	None	Progress callback

Returns CrawlResult:

• result.pages — list of ScrapeResult (url, content, status)
• result.failed — list of failed pages
• result.total_pages — total scraped
• result.total_failed — total failed
• result.to_text() — all content as single text string

---

AsyncIntelliScrape — Async Scraping

import asyncio
from intelliscrape import AsyncIntelliScrape

async def main():
    async with AsyncIntelliScrape() as scraper:
        urls = [
            "https://example.com",
            "https://python.org",
            "https://github.com",
        ]
        results = await scraper.scrape_many(urls, max_concurrent=5)
        for r in results:
            print(f"{r['url']}: {len(r['content'])} chars")

asyncio.run(main())

Standalone async functions:

from intelliscrape import scrape_async, scrape_many_async

result = await scrape_async("https://example.com")
results = await scrape_many_async(urls, max_concurrent=10)

---

DataExporter — Export Formats

from intelliscrape import DataExporter

DataExporter.to_json(data, file="output.json")
DataExporter.to_csv(data, file="output.csv")
DataExporter.to_excel(data, file="output.xlsx")
DataExporter.to_sqlite(data, file="output.db", table="scraped_data")
DataExporter.to_text(data, file="output.txt")
DataExporter.to_markdown(data, file="output.md")
DataExporter.export(data, format="json", file="output.json")

---

mirror() — Website Mirroring

Download entire websites for offline browsing with URL rewriting, robots.txt compliance, and archive support.

from intelliscrape import SiteMirror, MirrorConfig

# Quick mirror
from intelliscrape import mirror_site
result = mirror_site("https://example.com", max_depth=3)

mirror() — Convenience Function

from intelliscrape import mirror_site

result = mirror_site(
    url="https://example.com",
    output_dir="./mirror",
    max_depth=5,
    save_zip="site.zip",
    save_warc="archive.warc.gz",
)

Parameter	Type	Default	Description
url	str	required	Starting URL
output_dir	str	./mirror	Output directory
max_depth	int	5	Max link-following depth
save_zip	str	None	Create ZIP archive at path
save_warc	str	None	Create WARC archive at path
exclude_patterns	list	[]	URL exclude patterns
include_patterns	list	[]	URL include patterns
delay	float	0.5	Delay between requests (seconds)
respect_robots	bool	True	Respect robots.txt
engine	str	static	Scraping engine
proxy	str	None	Proxy URL

Returns MirrorResult:

• result.pages_downloaded — Number of HTML pages
• result.assets_downloaded — Number of assets (CSS, JS, images)
• result.total_bytes — Total bytes downloaded
• result.elapsed_seconds — Time taken
• result.errors — Number of errors
• result.output_dir — Output directory path
• result.zip_path — ZIP archive path (if created)
• result.warc_path — WARC archive path (if created)

SiteMirror — Full Control

from intelliscrape.track import SiteMirror, MirrorConfig

config = MirrorConfig(
    url="https://example.com",
    max_depth=3,
    output_dir="./my-mirror",
    exclude_patterns=["*.pdf", "/admin/*"],
    engine="static",
    delay=0.5,
    respect_robots=True,
    url_mode="relative",  # relative | absolute | keep_original
)

m = SiteMirror(config)
result = m.run(save_zip="mirror.zip", save_warc="mirror.warc.gz")

MirrorConfig — All Options

from intelliscrape.track import MirrorConfig

config = MirrorConfig(
    # What to mirror
    url="https://example.com",
    max_depth=5,
    max_pages=10000,
    max_file_size=50 * 1024 * 1024,  # 50 MB

    # Scope
    travel="same_domain",  # same_address | same_domain | same_tld | everywhere

    # What to fetch
    fetch_html=True,
    fetch_css=True,
    fetch_js=True,
    fetch_images=True,
    fetch_fonts=True,
    fetch_media=True,
    fetch_documents=True,

    # Filtering
    include_patterns=[],
    exclude_patterns=["*.pdf"],

    # Output
    output_dir="./mirror",
    url_mode="relative",
    generate_index=True,

    # Resume
    use_cache=True,
    update_mode=False,

    # Politeness
    delay=0.5,
    max_concurrent=5,
    respect_robots=True,

    # Engine & proxy
    engine="static",
    proxy=None,
    cookies=None,
)

CLI Examples

# Basic mirror
intelliscrape https://example.com --mirror

# Depth 3, custom output
intelliscrape https://example.com --mirror --mirror-depth 3 --mirror-output ./site

# With ZIP
intelliscrape https://example.com --mirror --mirror-zip backup.zip

# With WARC (web archive format)
intelliscrape https://example.com --mirror --mirror-warc archive.warc.gz

# With proxy
intelliscrape https://example.com --mirror --mirror-proxy socks5://proxy:1080

# Exclude patterns
intelliscrape https://example.com --mirror --mirror-exclude "*.pdf" --mirror-exclude "/api/*"

# Resume interrupted mirror
intelliscrape https://example.com --mirror --mirror-update

---

Downloader — File Downloads

from intelliscrape import Downloader

downloader = Downloader()

# Download linked files
results = downloader.download_links(html, base_url, "downloads/")

# Download all images
results = downloader.download_images(html, base_url, "downloads/images/")

---

Authenticator — Login & Sessions

from intelliscrape import Authenticator, LoginCredentials

auth = Authenticator()
credentials = LoginCredentials(
    username="user@example.com",
    password="secret",
)
success = auth.login("https://site.com/login", credentials)

---

FormSubmitter — Form Interaction

from intelliscrape import FormSubmitter

form_submitter = FormSubmitter()
forms = form_submitter.find_forms(html, base_url="https://site.com")
result_html = form_submitter.search(html, "query", base_url="https://site.com")

---

Paginator — Auto-pagination

from intelliscrape import Paginator

paginator = Paginator()
next_url = paginator.find_next_page(html, current_url, current_page)

---

RequestInterceptor — Request/Response Modification

from intelliscrape import RequestInterceptor

interceptor = RequestInterceptor()
interceptor.block_urls(["analytics", "tracking"])
interceptor.modify_headers({"X-Custom": "value"})
interceptor.add_response_handler(my_handler)

---

CookieManager — Cookie Persistence

from intelliscrape import CookieManager

cookie_mgr = CookieManager()
cookie_mgr.save_cookies("https://site.com", {"session": "abc123"})
cookies = cookie_mgr.load_cookies("https://site.com")

---

CaptchaDetector & CaptchaSolver

from intelliscrape import CaptchaDetector, CaptchaSolver

# Detect
captcha = CaptchaDetector.detect(html, url="https://site.com")

# Solve (requires API key)
solver = CaptchaSolver(provider="capsolver", api_key="YOUR_KEY")
token = solver.solve_recaptcha_v2(site_key, page_url)
token = solver.solve_hcaptcha(site_key, page_url)
token = solver.solve_turnstile(site_key, page_url)

---

AntiBotDetector — Anti-bot Vendor Detection

from intelliscrape import AntiBotDetector

info = AntiBotDetector.detect(html=html, headers=headers, cookies=cookies)
if info:
    print(info.vendor)       # AntiBotVendor.CLOUDFLARE
    print(info.confidence)   # 0.95

---

Anti-bot Bypass Classes

from intelliscrape import (
    CloudflareTurnstileBypass,
    DataDomeBypass,
    PerimeterXBypass,
    AkamaiBypass,
)

Each bypass class provides detection, recommended settings, and automated token solving where possible.

---

Engine System

IntelliScrape uses a 4-tier engine escalation system. It tries the cheapest, fastest method first and escalates only when needed.

Tier 1: Static (curl_cffi)        → Sub-second, TLS impersonation
    ↓ if JS-only content
Tier 2: Playwright Stealth         → 2-5s, headless Chromium + patches
    ↓ if still blocked
Tier 3: Camoufox                   → 3-8s, custom Firefox (C++ patches)
    ↓ if still blocked
Tier 4: nodriver                   → 5-15s, raw CDP, no WebDriver traces

Tier	Engine	Speed	Stealth	Best For
1	static	Sub-second	Low	Static sites, APIs
2	playwright_stealth	2-5s	Medium	JS-heavy sites, basic bot detection
3	camoufox	3-8s	High	Protected sites, fingerprint detection
4	nodriver	5-15s	Maximum	DataDome, PerimeterX, Akamai

# Auto-detect (default)
text = scraper.scrape("https://site.com")

# Force specific engine
text = scraper.scrape("https://site.com", engine="playwright_stealth")

# Force browser for known JS-heavy sites
text = scraper.scrape("https://react-app.com", force_browser=True)

---

Intelligent Mode

Enabled by default (intelligent=True). Before scraping, IntelliScrape analyzes the URL to determine:

• Site type — ecommerce, social, news, tech, education, etc.
• Protection level — none, basic, moderate, high, extreme
• Recommended engine — which tier to start with
• Recommended delay — slower for protected sites
• Residential proxy needed — auto-selects proxy type

analysis = scraper.analyze("https://amazon.com")
print(analysis.site_type.value)         # "ecommerce"
print(analysis.protection_level.value)  # "high"
print(analysis.recommended_engine)      # "playwright_stealth"
print(analysis.requires_residential_proxy)  # True

---

Features

Anti-Detection

Feature	Description
TLS Fingerprinting	Impersonates Chrome, Firefox, Safari (JA3/JA4)
Header Rotation	Randomizes HTTP headers
Browser Fingerprinting	Randomizes viewport, timezone, WebGL, canvas
Human Simulation	Bezier mouse paths, natural scrolls, realistic delays
Cookie Consent	Auto-handles consent banners
Rate Limiting	Smart delays based on site protection
Retry with Backoff	Exponential backoff on failures

CAPTCHA Solving

Automated (requires API key):

scraper = IntelliScrape(api_key="YOUR_KEY", captcha_provider="capsolver")
result = scraper.scrape("https://protected-site.com")

CAPTCHA Type	2Captcha	CapSolver
reCAPTCHA v2	Yes	Yes
reCAPTCHA v3	No	Yes
hCaptcha	Yes	Yes
Cloudflare Turnstile	No	Yes

Manual (opens visible browser):

scraper = IntelliScrape(manual_captcha=True)
result = scraper.scrape("https://site-with-captcha.com")
# Browser opens → solve CAPTCHA → press Enter in terminal

intelliscrape https://site.com --manual-captcha

Proxy Configuration

# Single proxy
scraper = IntelliScrape(proxy="user:pass@proxy:8080")

# Multiple proxies
scraper = IntelliScrape(proxies=["proxy1:8080", "proxy2:8080"])

# Residential proxy
scraper = IntelliScrape(brightdata_key="YOUR_KEY")

# Free proxies (automatic)
scraper = IntelliScrape(use_free_proxies=True)

Export Formats

from intelliscrape import DataExporter

DataExporter.to_json(data, file="output.json")
DataExporter.to_csv(data, file="output.csv")
DataExporter.to_excel(data, file="output.xlsx")
DataExporter.to_sqlite(data, file="output.db")
DataExporter.to_markdown(data, file="output.md")

intelliscrape https://site.com --export csv -o data.csv
intelliscrape https://site.com --export json -o data.json

Website Mirroring

Download complete websites for offline browsing with URL rewriting and archive support.

from intelliscrape import mirror_site

# Basic mirror
result = mirror_site("https://example.com", max_depth=3)

# With ZIP archive
result = mirror_site("https://example.com", save_zip="site.zip")

# Full options
from intelliscrape.track import SiteMirror, MirrorConfig

config = MirrorConfig(
    url="https://example.com",
    max_depth=3,
    output_dir="./mirror",
    exclude_patterns=["*.pdf", "/admin/*"],
    engine="static",
    delay=0.5,
)
m = SiteMirror(config)
result = m.run(save_zip="mirror.zip", save_warc="mirror.warc.gz")

# Mirror site
intelliscrape https://example.com --mirror

# Mirror with depth and output dir
intelliscrape https://example.com --mirror --mirror-depth 3 --mirror-output ./backup

# Mirror + ZIP
intelliscrape https://example.com --mirror --mirror-zip backup.zip

# Mirror + WARC (web archive format)
intelliscrape https://example.com --mirror --mirror-warc archive.warc.gz

# Mirror with proxy
intelliscrape https://example.com --mirror --mirror-proxy socks5://proxy:1080

---

Examples

Scrape React/Vue/Angular SPAs

result = scraper.scrape("https://react-app.com", force_browser=True)

Scrape with Custom Headers

result = scraper.scrape(
    "https://api.example.com/data",
    headers={"Authorization": "Bearer token123"},
)

Persistent Sessions

scraper = IntelliScrape(session_profile="my_session")
scraper.scrape("https://site.com")       # Creates session
scraper.scrape("https://site.com/dashboard")  # Reuses session

Download Files

from intelliscrape import Downloader

downloader = Downloader()
html = scraper.scrape("https://example.com/downloads", return_raw=True)
results = downloader.download_links(html, "https://example.com", "downloads/")

Batch Scraping with Export

from intelliscrape import IntelliScrape, DataExporter

scraper = IntelliScrape()
urls = [f"https://example.com/page/{i}" for i in range(100)]

results = scraper.scrape_many(urls)
DataExporter.to_csv(
    [{"url": r["url"], "content": r["content"], "success": r["success"]} for r in results],
    file="results.csv",
)

Check Links on a Page

from intelliscrape import check_links

report = check_links("https://example.com")

# Summary
print(f"Total: {report.summary.total}")
print(f"OK: {report.summary.ok}")
print(f"Broken: {report.summary.broken}")
print(f"Success rate: {report.summary.success_rate:.1f}%")

# Only internal links
report = check_links("https://example.com", ignore_external=True)

# Export broken links
for link in report.links:
    if not link.is_ok:
        print(f"BROKEN: {link.url} -> {link.status_code}")

---

Troubleshooting

Problem	Solution
Returns empty or widget text	Use force_browser=True — site is a JS SPA
CAPTCHA blocking	Use manual_captcha=True or api_key + captcha_provider
Blocked by anti-bot	Try engine="camoufox" + residential proxy
Playwright not installed	pip install playwright && playwright install chromium
Camoufox not installed	pip install camoufox && camoufox install
nodriver not installed	pip install nodriver

---

Project Structure

intelliscrape/
    __init__.py             # Public API exports
    __main__.py             # python -m intelliscrape
    core.py                 # IntelliScrape class (main orchestrator)
    cli.py                  # CLI (argparse + rich)
    async_scraper.py        # AsyncIntelliScrape
    intelligent.py          # SiteAnalyzer, SmartRateLimiter
    auth.py                 # Authenticator, LoginCredentials
    forms.py                # FormSubmitter
    pagination.py           # Paginator
    export.py               # DataExporter
    downloader.py           # Downloader
    cookies.py              # CookieManager
    crawler.py              # crawl(), CrawlResult
    interceptor.py          # RequestInterceptor
    link_checker.py         # check_links, LinkCheckReport
    parser.py               # HTML DOM builder
    cleaner.py              # Text cleaning
    utils.py                # HTML analysis
    exceptions.py           # Exceptions
    retry.py                # SmartRetry
    ip_manager.py           # IPManager, NaturalRotator

    engines/                # 4-tier scraping engines
        base.py             # BaseEngine, ScrapeResult
        static.py           # curl_cffi (Tier 1)
        playwright_stealth.py  # Playwright (Tier 2)
        camoufox.py         # Camoufox (Tier 3)
        stealth.py          # nodriver (Tier 4)

    anti_detection/         # Anti-detection subsystem
        antibot.py          # AntiBotDetector
        behavior.py         # HumanBehavior
        bypass.py           # Vendor-specific bypasses
        consent.py          # CookieConsentHandler
        fingerprint.py      # FingerprintGenerator
        headers.py          # HeaderManager
        throttle.py         # SmartThrottle, RateLimiter
        tls.py              # TLSConfig (JA3/JA4)

    challenges/             # Challenge handling
        captcha.py          # CaptchaDetector, CaptchaSolver

    extractor/              # Content extraction
        structured.py       # StructuredExtractor, StructuredData

    proxy/                  # Proxy management
        __init__.py         # ProxyConfig, ProxyManager
        free_finder.py      # FreeProxyFinder
        manager.py          # IntelligentProxyManager
        providers.py        # BrightData, ScraperAPI, etc.

    session/                # Session persistence
        __init__.py         # SessionManager

    track/                  # Website mirroring (HTTrack port)
        __init__.py         # Package exports
        config.py           # MirrorConfig (30+ options)
        mirror.py           # SiteMirror engine (async workers, WARC/ZIP)
        parser.py           # AssetDiscovery (HTML/CSS/JS extraction)
        rewriter.py         # URLRewriter (relative/absolute paths)
        naming.py           # SaveNamer (URL→filesystem mapping)
        cache.py            # MirrorCache (resume support)
        filters.py          # URLFilter (include/exclude patterns)
        robots.py           # RobotsParser (RFC 9309 compliance)

---

Contributing

We welcome contributions! See CONTRIBUTING.md.

git clone https://github.com/GuixJoy/IntelliScrape.git
cd IntelliScrape/IntelliScrape_library
pip install -e ".[dev]"
pytest

---

License

MIT License — see LICENSE.

---