IntelliScrape

Scrape anything. Nothing scrapes back.

IntelliScrape is a Python web scraping library with anti-detection, TLS fingerprint impersonation, and stealth browsing. It uses a 4-tier engine system that automatically escalates from fast HTTP requests to full browser automation — so you get the cheapest, fastest method that works, and heavier weapons only when needed.

Installation
Quick Start
CLI Reference
Library API Reference
Engine System
Intelligent Mode
Anti-Detection
CAPTCHA Solving
Manual CAPTCHA Solving
Proxy Configuration
Export Formats
Async Support
Advanced Usage
Troubleshooting
Contributing
License

Installation

pip install intelliscrape

Optional Extras

# Stealth browsing (nodriver engine)
pip install intelliscrape[stealth]

# Camoufox engine (maximum stealth, Firefox-based)
pip install intelliscrape[camoufox]

# CAPTCHA solving (2Captcha, CapSolver)
pip install intelliscrape[captcha]

# Async support (concurrent scraping)
pip install intelliscrape[async]

# Everything
pip install intelliscrape[all]

# Development
pip install intelliscrape[dev]

System Requirements

Python 3.9+
For browser engines: playwright install chromium (or camoufox install for Camoufox)

Quick Start

CLI

# Scrape any website
intelliscrape https://example.com

# Save output to file
intelliscrape https://example.com -o output.txt

# Get structured JSON (title, description, meta tags)
intelliscrape https://example.com --json

# Analyze a site (see what approach IntelliScrape would use)
intelliscrape https://amazon.com --analyze

# Crawl entire website
intelliscrape https://docs.python.org --crawl --max-pages 50

Python — One-liner

from intelliscrape import scrape

text = scrape("https://news.ycombinator.com")
print(text[:500])

Python — Full-featured class

from intelliscrape import IntelliScrape

scraper = IntelliScrape()
result = scraper.scrape("https://example.com")
print(result)

Get structured data

from intelliscrape import IntelliScrape

scraper = IntelliScrape()
data = scraper.get_structured("https://github.com")

print(data.title)          # Page title
print(data.description)    # Meta description
print(data.og_data)        # OpenGraph tags
print(data.json_ld)        # JSON-LD structured data

Crawl entire website

from intelliscrape import crawl

result = crawl("https://docs.python.org", max_pages=100)
print(f"Scraped {result.total_pages} pages, {result.total_failed} failed")

for page in result.pages:
    print(f"  {page.url}: {len(page.content)} chars")

CLI Reference

intelliscrape [URL] [OPTIONS]

Output Options

Flag	Description
`-o, --output FILE`	Save output to file
`--json`	Output structured JSON (title, description, meta tags, etc.)
`--raw`	Output raw HTML instead of extracted text

Intelligent Mode

Flag	Description
`--analyze`	Analyze site and show recommendations (no scraping)
`--no-intelligent`	Disable intelligent auto-detection mode

Engine Control

Flag	Description
`--force-browser`	Force browser engine for JS-heavy sites
`--manual-captcha`	Open a visible browser when CAPTCHA detected, wait for user to solve

Proxy Options

Flag	Description
`--use-free-proxies`	Use free proxies automatically
`--no-free-proxies`	Disable free proxy finder
`--find-proxies`	Find and test free proxies (no URL needed)
`--brightdata-key KEY`	Bright Data API key for residential proxies
`--scraperapi-key KEY`	ScraperAPI key
`--oxylabs-key KEY`	Oxylabs API key
`--smartproxy-key KEY`	Smartproxy API key

Authentication

Flag	Description
`--login`	Login to site before scraping
`--username USER`	Login username/email
`--password PASS`	Login password
`--login-url URL`	Explicit login URL

Cookies

Flag	Description
`--save-cookies FILE`	Save cookies to JSON file
`--load-cookies FILE`	Load cookies from JSON file

Request Modification

Flag	Description
`--block PATTERNS`	Block URLs (comma-separated patterns)
`--header "Key: Value"`	Add custom header (repeatable)

Pagination & Search

Flag	Description
`--paginate`	Auto-follow pagination links
`--max-pages N`	Max pages for crawl/pagination (default: 50)
`--search QUERY`	Submit search query on the page

Crawl

Flag	Description
`--crawl`	Crawl entire website

Downloads

Flag	Description
`--download`	Download linked files from page
`--download-images`	Download all images from page
`--download-dir DIR`	Download directory (default: "downloads")

Export

Flag	Description
`--export FORMAT`	Export format: `json`, `csv`, `excel`, `sqlite`, `text`, `markdown`

CLI Examples

# Basic scraping
intelliscrape https://example.com
intelliscrape https://example.com -o output.txt
intelliscrape https://example.com --json

# Analyze site protection
intelliscrape https://amazon.com --analyze

# Find free proxies
intelliscrape --find-proxies

# Use free proxies
intelliscrape https://amazon.com --use-free-proxies

# Login and scrape
intelliscrape https://site.com --login --username user --password pass

# Save/load cookies
intelliscrape https://site.com --save-cookies cookies.json
intelliscrape https://site.com --load-cookies cookies.json

# Custom headers
intelliscrape https://site.com --header "Authorization: Bearer xxx"

# Pagination
intelliscrape https://example.com/products --paginate --max-pages 10

# Search
intelliscrape https://google.com --search "python scraping"

# Download files
intelliscrape https://example.com --download
intelliscrape https://example.com --download-images

# Export formats
intelliscrape https://example.com --export csv -o data.csv
intelliscrape https://example.com --export json -o data.json

# Crawl entire site
intelliscrape https://docs.python.org --crawl --max-pages 50

# Force browser for JS-heavy sites
intelliscrape https://react-app.com --force-browser

# Manual CAPTCHA solving
intelliscrape https://protected-site.com --manual-captcha

# With residential proxy
intelliscrape https://amazon.com --brightdata-key YOUR_KEY

Library API Reference

`scrape()` — Quick One-liner

from intelliscrape import scrape

text = scrape(url, **kwargs)

Parameter	Type	Default	Description
`url`	str	required	Target URL
`engine`	str	None	Force engine: `"static"`, `"playwright_stealth"`, `"camoufox"`, `"nodriver"`
`extract`	bool	True	Extract text from HTML
`clean`	bool	True	Clean extracted text
`return_raw`	bool	False	Return raw HTML
`return_structured`	bool	False	Return `StructuredData` object
`handle_consent`	bool	True	Handle cookie consent banners
`force_browser`	bool	False	Force browser engine

`IntelliScrape` — Main Class

from intelliscrape import IntelliScrape

scraper = IntelliScrape(**kwargs)

Constructor Parameters

Parameter	Type	Default	Description
`proxy`	str, ProxyConfig, or list	None	Single proxy or list of proxies
`proxies`	list of str	None	Proxy strings (`host:port` or `user:pass@host:port`)
`brightdata_key`	str	None	Bright Data API key
`scraperapi_key`	str	None	ScraperAPI key
`oxylabs_key`	str	None	Oxylabs API key
`smartproxy_key`	str	None	Smartproxy API key
`prefer_residential`	bool	True	Prefer residential proxies
`use_free_proxies`	bool	True	Auto-find free proxies if none provided
`api_key`	str	None	CAPTCHA solving API key (2Captcha/CapSolver)
`captcha_provider`	str	None	`"2captcha"` or `"capsolver"`
`headless`	bool	True	Run browser in headless mode
`simulate_behavior`	bool	True	Enable human-like behavioral simulation
`manual_captcha`	bool	False	Open visible browser for manual CAPTCHA solving
`tls_profile`	str	`"chrome131"`	TLS fingerprint profile to impersonate
`session_profile`	str	None	Persistent session profile name
`max_retries`	int	3	Maximum retry attempts
`min_delay`	float	0.5	Minimum delay between requests (seconds)
`max_delay`	float	3.0	Maximum delay between requests (seconds)
`requests_per_minute`	int	None	Rate limit (requests per minute)
`intelligent`	bool	True	Enable intelligent auto-detection
`log_level`	str	`"WARNING"`	Logging level

Methods

`scrape(url, **kwargs)`

Scrape a URL and return text content.

result = scraper.scrape(
    url="https://example.com",
    engine=None,            # Force specific engine
    extract=True,           # Extract text
    clean=True,             # Clean text
    return_raw=False,       # Return raw HTML
    return_structured=False, # Return StructuredData
    handle_consent=True,    # Handle cookie consent
    force_browser=False,    # Force browser engine
    intelligent=None,       # Override intelligent mode
)

`get_structured(url, **kwargs)`

Get structured data (title, description, meta tags, JSON-LD).

data = scraper.get_structured("https://github.com")
print(data.title)
print(data.description)
print(data.og_data)
print(data.json_ld)

`analyze(url)`

Analyze a site and return recommendations.

analysis = scraper.analyze("https://amazon.com")
print(analysis.site_type)           # "ecommerce"
print(analysis.protection_level)    # "high"
print(analysis.recommended_engine)  # "playwright_stealth"
print(analysis.recommended_delay)   # 3.0

`scrape_many(urls, **kwargs)`

Scrape multiple URLs with rate limiting.

results = scraper.scrape_many([
    "https://example.com/page1",
    "https://example.com/page2",
])
# Returns: [{"url": ..., "content": ..., "success": ..., "error": ...}, ...]

`check_captcha(url)`

Check if a URL has a CAPTCHA.

captcha = scraper.check_captcha("https://site.com")
if captcha:
    print(captcha.captcha_type)  # "recaptcha_v2", "hcaptcha", etc.
    print(captcha.site_key)

`check_antibot(url)`

Check anti-bot protection on a URL.

info = scraper.check_antibot("https://site.com")
if info:
    print(info.vendor)       # "cloudflare", "akamai", etc.
    print(info.confidence)   # 0.95

`find_free_proxies(test=True)`

Find and test free proxies.

proxies = scraper.find_free_proxies(test=True)
for p in proxies:
    print(f"{p['url']} - speed: {p['speed']:.2f}s")

`get_proxy_status()`

Get proxy manager status.

status = scraper.get_proxy_status()
print(status['user_proxies'])
print(status['healthy_proxies'])
print(status['providers_available'])

`crawl()` — Website Crawler

from intelliscrape import crawl

result = crawl(
    url="https://docs.python.org",
    max_pages=50,
    delay=0.5,
    on_page=None,  # Callback: on_page(done, failed)
)

Parameter	Type	Default	Description
`url`	str	required	Starting URL
`max_pages`	int	50	Maximum pages to crawl
`delay`	float	0.5	Delay between requests (seconds)
`on_page`	callable	None	Progress callback `on_page(done, failed)`

Returns CrawlResult with:

result.pages — list of ScrapeResult (url, content, status)
result.failed — list of failed pages
result.total_pages — total scraped
result.total_failed — total failed
result.to_text() — all content as single text string

`AsyncIntelliScrape` — Async Scraping

import asyncio
from intelliscrape import AsyncIntelliScrape

async def main():
    async with AsyncIntelliScrape() as scraper:
        urls = [
            "https://example.com",
            "https://python.org",
            "https://github.com",
        ]
        results = await scraper.scrape_many(urls, max_concurrent=5)
        for r in results:
            print(f"{r['url']}: {len(r['content'])} chars")

asyncio.run(main())

Also available as standalone functions:

from intelliscrape import scrape_async, scrape_many_async

# Single URL
result = await scrape_async("https://example.com")

# Multiple URLs
results = await scrape_many_async(urls, max_concurrent=10)

`DataExporter` — Export Formats

from intelliscrape import DataExporter

# JSON
DataExporter.to_json(data, file="output.json")

# CSV
DataExporter.to_csv(data, file="output.csv")

# Excel
DataExporter.to_excel(data, file="output.xlsx")

# SQLite
DataExporter.to_sqlite(data, file="output.db", table="scraped_data")

# Text
DataExporter.to_text(data, file="output.txt")

# Markdown
DataExporter.to_markdown(data, file="output.md")

# Generic export
DataExporter.export(data, format="json", file="output.json")

`Downloader` — File Downloads

from intelliscrape import Downloader

downloader = Downloader()

# Download all linked files
results = downloader.download_links(html, base_url, "downloads/")

# Download all images
results = downloader.download_images(html, base_url, "downloads/images/")

for r in results:
    print(f"{'OK' if r.success else 'FAIL'}: {r.url}")

`Authenticator` — Login & Sessions

from intelliscrape import Authenticator, LoginCredentials

auth = Authenticator()
credentials = LoginCredentials(
    username="user@example.com",
    password="secret",
)

success = auth.login(
    "https://site.com/login",
    credentials,
    login_url="https://site.com/login",  # optional
)

`FormSubmitter` — Form Interaction

from intelliscrape import FormSubmitter

form_submitter = FormSubmitter()

# Find forms
forms = form_submitter.find_forms(html, base_url="https://site.com")

# Submit search
result_html = form_submitter.search(html, "python scraping", base_url="https://site.com")

`Paginator` — Auto-pagination

from intelliscrape import Paginator

paginator = Paginator()

# Find next page link
next_url = paginator.find_next_page(html, current_url, current_page)

`RequestInterceptor` — Request/Response Modification

from intelliscrape import RequestInterceptor

interceptor = RequestInterceptor()

# Block analytics URLs
interceptor.block_urls(["analytics", "tracking"])

# Add custom headers
interceptor.modify_headers({"X-Custom": "value"})

# Add response handler
def my_handler(response):
    response.body = response.body.replace("old", "new")
    return response

interceptor.add_response_handler(my_handler)

`CookieManager` — Cookie Persistence

from intelliscrape import CookieManager

cookie_mgr = CookieManager()

# Save cookies
cookie_mgr.save_cookies("https://site.com", {"session": "abc123"})

# Load cookies
cookies = cookie_mgr.load_cookies("https://site.com")

`CaptchaDetector` & `CaptchaSolver` — CAPTCHA Handling

from intelliscrape import CaptchaDetector, CaptchaSolver

# Detect CAPTCHA
captcha = CaptchaDetector.detect(html, url="https://site.com")
if captcha:
    print(captcha.captcha_type)  # CaptchaType.RECAPTCHA_V2
    print(captcha.site_key)

# Solve CAPTCHA (requires API key)
solver = CaptchaSolver(provider="capsolver", api_key="YOUR_KEY")
token = solver.solve_recaptcha_v2(site_key, page_url)
token = solver.solve_hcaptcha(site_key, page_url)
token = solver.solve_turnstile(site_key, page_url)

`AntiBotDetector` — Anti-bot Vendor Detection

from intelliscrape import AntiBotDetector

info = AntiBotDetector.detect(html=html, headers=headers, cookies=cookies)
if info:
    print(info.vendor)       # AntiBotVendor.CLOUDFLARE
    print(info.confidence)   # 0.95
    print(info.indicators)   # ["cf-browser-verification", ...]

Anti-bot Bypass Classes

from intelliscrape import (
    CloudflareTurnstileBypass,
    DataDomeBypass,
    PerimeterXBypass,
    AkamaiBypass,
)

# Each bypass class provides:
# - Detection of the anti-bot vendor
# - Recommended engine, proxy, and behavior settings
# - Automated token solving (where possible)

Engine System — 4-Tier Escalation

IntelliScrape uses a tiered engine system. It tries the cheapest, fastest method first and escalates only when needed.

scrape(url)
    |
    v
+---------------------------+
| Tier 1: Static (curl_cffi)|
| TLS impersonation         |
| Sub-second                |
+---------------------------+
    | if JS-only content
    v
+-------------------------------+
| Tier 2: Playwright Stealth    |
| Headless Chromium + patches   |
| JS rendering                  |
+-------------------------------+
    | if still blocked
    v
+-------------------------------+
| Tier 3: Camoufox             |
| Custom Firefox (C++ patches) |
| Maximum stealth              |
+-------------------------------+
    | if still blocked
    v
+-------------------------------+
| Tier 4: nodriver             |
| Raw CDP, no WebDriver traces |
+-------------------------------+

Tier	Engine	Speed	Stealth	Best For
1	`static` (curl_cffi)	Sub-second	Low	Static sites, APIs
2	`playwright_stealth`	2-5s	Medium	JS-heavy sites, basic bot detection
3	`camoufox`	3-8s	High	Protected sites, fingerprint detection
4	`nodriver`	5-15s	Maximum	DataDome, PerimeterX, Akamai

# Auto-detect (default)
text = scraper.scrape("https://site.com")

# Force specific engine
text = scraper.scrape("https://site.com", engine="playwright_stealth")

# Force browser for known JS-heavy sites
text = scraper.scrape("https://react-app.com", force_browser=True)

Intelligent Mode

Enabled by default (intelligent=True). Before scraping, IntelliScrape analyzes the URL to determine:

Site type (ecommerce, social, news, tech, education, etc.)
Protection level (none, basic, moderate, high, extreme)
Recommended engine (which tier to start with)
Recommended delay (slower for protected sites)
Whether residential proxy is needed

# Analyze a site
analysis = scraper.analyze("https://amazon.com")
print(analysis.site_type.value)         # "ecommerce"
print(analysis.protection_level.value)  # "high"
print(analysis.recommended_engine)      # "playwright_stealth"
print(analysis.recommended_delay)       # 3.0
print(analysis.requires_residential_proxy)  # True

# Disable intelligent mode
text = scraper.scrape("https://site.com", intelligent=False)

Anti-Detection

IntelliScrape includes multiple layers of anti-detection:

Feature	Description
TLS Fingerprinting	Impersonates Chrome, Firefox, Safari TLS fingerprints (JA3/JA4)
Header Rotation	Randomizes HTTP headers to avoid fingerprinting
Browser Fingerprinting	Randomizes viewport, timezone, language, WebGL, canvas
Human Simulation	Bezier curve mouse movements, natural scroll patterns, realistic delays
Cookie Consent	Auto-detects and handles cookie consent banners
Rate Limiting	Smart delays based on site protection level
Retry with Backoff	Exponential backoff with jitter on failures

# Disable behavior simulation
scraper = IntelliScrape(simulate_behavior=False)

# Custom TLS profile
scraper = IntelliScrape(tls_profile="firefox120")

# Custom rate limiting
scraper = IntelliScrape(
    min_delay=1.0,
    max_delay=5.0,
    requests_per_minute=20,
)

CAPTCHA Solving

Automated (via API)

Requires an API key from 2Captcha or CapSolver.

from intelliscrape import IntelliScrape

scraper = IntelliScrape(
    api_key="YOUR_API_KEY",
    captcha_provider="capsolver",  # or "2captcha"
)

# CAPTCHA solving is triggered automatically when detected
result = scraper.scrape("https://protected-site.com")

Supported CAPTCHA types:

Type	2Captcha	CapSolver
reCAPTCHA v2	Yes	Yes
reCAPTCHA v3	No	Yes
hCaptcha	Yes	Yes
Cloudflare Turnstile	No	Yes
FunCaptcha	No	No

Manual CAPTCHA Solving

When manual_captcha=True, IntelliScrape opens a visible browser window if a CAPTCHA is detected, waits for you to solve it, then continues scraping.

from intelliscrape import IntelliScrape

scraper = IntelliScrape(manual_captcha=True)
result = scraper.scrape("https://site-with-captcha.com")
# A browser window opens -> solve CAPTCHA -> press Enter in terminal

# CLI
intelliscrape https://site-with-captcha.com --manual-captcha

Proxy Configuration

Single Proxy

scraper = IntelliScrape(proxy="user:pass@proxy:8080")

Multiple Proxies

scraper = IntelliScrape(proxies=[
    "user:pass@proxy1:8080",
    "user:pass@proxy2:8080",
])

Residential Proxy Providers

scraper = IntelliScrape(
    brightdata_key="YOUR_BRIGHTDATA_KEY",
    # scraperapi_key="YOUR_KEY",
    # oxylabs_key="YOUR_KEY",
    # smartproxy_key="YOUR_KEY",
    prefer_residential=True,
)

Free Proxies (Automatic)

scraper = IntelliScrape(use_free_proxies=True)  # default
text = scraper.scrape("https://site.com")

# CLI
intelliscrape https://site.com --use-free-proxies
intelliscrape --find-proxies  # Just find proxies, no scraping

Export Formats

CLI

intelliscrape https://site.com --export json -o data.json
intelliscrape https://site.com --export csv -o data.csv
intelliscrape https://site.com --export excel -o data.xlsx
intelliscrape https://site.com --export sqlite -o data.db
intelliscrape https://site.com --export text -o data.txt
intelliscrape https://site.com --export markdown -o data.md

Python

from intelliscrape import DataExporter

data = [
    {"url": "https://example.com", "title": "Example", "content": "..."},
    {"url": "https://python.org", "title": "Python", "content": "..."},
]

DataExporter.to_json(data, file="output.json")
DataExporter.to_csv(data, file="output.csv")
DataExporter.to_excel(data, file="output.xlsx")
DataExporter.to_sqlite(data, file="output.db", table="pages")
DataExporter.to_markdown(data, file="output.md")

Async Support

import asyncio
from intelliscrape import AsyncIntelliScrape

async def main():
    async with AsyncIntelliScrape(
        proxy="user:pass@proxy:8080",
        headless=True,
        max_concurrent=10,
    ) as scraper:
        urls = [f"https://example.com/page/{i}" for i in range(20)]
        results = await scraper.scrape_many(urls)
        for r in results:
            if r["success"]:
                print(f"{r['url']}: {len(r['content'])} chars")

asyncio.run(main())

Advanced Usage

Scrape with Login

from intelliscrape import IntelliScrape, Authenticator, LoginCredentials

scraper = IntelliScrape()
auth = Authenticator(scraper.session_manager.session)

# Login
credentials = LoginCredentials(username="user@email.com", password="pass")
auth.login("https://site.com/login", credentials)

# Now scrape authenticated pages
result = scraper.scrape("https://site.com/dashboard")

Scrape with Custom Headers

scraper = IntelliScrape()
result = scraper.scrape(
    "https://api.example.com/data",
    headers={"Authorization": "Bearer token123", "X-Custom": "value"},
)

Block URLs

from intelliscrape import RequestInterceptor

interceptor = RequestInterceptor()
interceptor.block_urls(["analytics", "tracking", "ads"])

result = scraper.scrape("https://site.com", interceptor=interceptor)

Scrape React/Vue/Angular SPAs

# Force browser engine for JavaScript-heavy SPAs
result = scraper.scrape("https://react-app.com", force_browser=True)

# Or force specific engine
result = scraper.scrape("https://vue-app.com", engine="playwright_stealth")

Persistent Sessions

scraper = IntelliScrape(session_profile="my_session")

# First run: creates session
scraper.scrape("https://site.com")

# Subsequent runs: reuses session cookies
scraper.scrape("https://site.com/dashboard")

Download Files

from intelliscrape import Downloader

downloader = Downloader()

# Download linked PDFs, ZIPs, etc.
html = scraper.scrape("https://example.com/downloads", return_raw=True)
results = downloader.download_links(html, "https://example.com", "downloads/")

# Download all images
results = downloader.download_images(html, "https://example.com", "downloads/images/")

Troubleshooting

Site returns empty or accessibility widget text

The site is likely a JavaScript SPA. Force browser mode:

result = scraper.scrape(url, force_browser=True)

Or via CLI:

intelliscrape https://site.com --force-browser

CAPTCHA blocking scraping

Use manual CAPTCHA solving:

scraper = IntelliScrape(manual_captcha=True)
result = scraper.scrape("https://protected-site.com")

Or automated solving:

scraper = IntelliScrape(api_key="YOUR_KEY", captcha_provider="capsolver")
result = scraper.scrape("https://protected-site.com")

Getting blocked by anti-bot

Try escalating engines:

# Try with maximum stealth
result = scraper.scrape(url, engine="camoufox")

# With residential proxy
scraper = IntelliScrape(brightdata_key="YOUR_KEY")
result = scraper.scrape(url)

Playwright not installed

pip install playwright
playwright install chromium

Camoufox not installed

pip install camoufox
camoufox install

nodriver not installed

pip install nodriver

Project Structure

intelliscrape/
    __init__.py             # Public API exports
    __main__.py             # Entry point for `python -m intelliscrape`
    core.py                 # IntelliScrape class — main orchestrator
    cli.py                  # CLI (argparse + rich output)
    async_scraper.py        # AsyncIntelliScrape, scrape_async, scrape_many_async
    intelligent.py          # SiteAnalyzer, SmartRateLimiter
    auth.py                 # Authenticator, LoginCredentials
    forms.py                # FormSubmitter, Form, FormField
    pagination.py           # Paginator, PageInfo
    export.py               # DataExporter (JSON, CSV, Excel, SQLite, Text, Markdown)
    downloader.py           # Downloader for images/files
    cookies.py              # CookieManager — persistent cookie storage
    crawler.py              # crawl() function, CrawlResult
    interceptor.py          # RequestInterceptor, ResponseModifier
    parser.py               # HTML DOM builder
    cleaner.py              # Text cleaning utilities
    utils.py                # HTML analysis utilities
    exceptions.py           # IntelliScrapeError, DownloadError
    retry.py                # SmartRetry with engine fallback
    ip_manager.py           # IPManager, NaturalRotator
    link_checker.py         # Link validation

    engines/                # Scraping engines (4-tier)
        base.py             # BaseEngine ABC, ScrapeResult dataclass
        static.py           # StaticEngine (curl_cffi — Tier 1)
        playwright_stealth.py  # PlaywrightStealthEngine (Tier 2)
        camoufox.py         # CamoufoxEngine (Tier 3)
        stealth.py          # StealthEngine (nodriver — Tier 4)

    anti_detection/         # Anti-detection subsystem
        antibot.py          # AntiBotDetector — vendor fingerprinting
        behavior.py         # HumanBehavior — mouse paths, scroll patterns
        bypass.py           # CloudflareTurnstileBypass, DataDomeBypass, etc.
        consent.py          # CookieConsentHandler
        fingerprint.py      # FingerprintGenerator
        headers.py          # HeaderManager
        throttle.py         # SmartThrottle, RateLimiter
        tls.py              # TLSConfig — JA3/JA4 impersonation

    challenges/             # Challenge handling
        captcha.py          # CaptchaDetector, CaptchaSolver

    extractor/              # Content extraction
        structured.py       # StructuredExtractor, StructuredData

    proxy/                  # Proxy management
        __init__.py         # ProxyConfig, ProxyManager
        free_finder.py      # FreeProxyFinder
        manager.py          # IntelligentProxyManager
        providers.py        # BrightDataProvider, ScraperAPIProvider, etc.

    session/                # Session persistence
        __init__.py         # SessionManager

Contributing

We welcome contributions! Whether it's:

New anti-bot bypass patterns
CAPTCHA solving techniques
Proxy provider integrations
Bug fixes
Documentation

See CONTRIBUTING.md to get started.

# Clone the repo
git clone https://github.com/GuixJoy/IntelliScrape.git
cd IntelliScrape/IntelliScrape_library

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run linting
ruff check intelliscrape/

Community

GitHub Issues: Report bugs
Discussions: Ask questions
PyPI: pypi.org/project/intelliscrape

License

MIT License — see LICENSE for details.

Built with for the data community.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

intelliscrape-2.5.0.tar.gz (113.2 kB view details)

Uploaded Jul 29, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

intelliscrape-2.5.0-py3-none-any.whl (115.6 kB view details)

Uploaded Jul 29, 2026 Python 3

File details

Details for the file intelliscrape-2.5.0.tar.gz.

File metadata

Download URL: intelliscrape-2.5.0.tar.gz
Upload date: Jul 29, 2026
Size: 113.2 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/6.2.0 CPython/3.12.10

File hashes

Hashes for intelliscrape-2.5.0.tar.gz
Algorithm	Hash digest
SHA256	`e27a174a0314ac9235029b6c581f0805611fd0a2c1f0a673ca5d0af41ed2f025`
MD5	`2aef52989ebb773ebad16193b4409379`
BLAKE2b-256	`ca06742f4dfc59e137ca87ccfd22389b1859823fffd6dc136fae0a213f48e1a0`

See more details on using hashes here.

File details

Details for the file intelliscrape-2.5.0-py3-none-any.whl.

File metadata

Download URL: intelliscrape-2.5.0-py3-none-any.whl
Upload date: Jul 29, 2026
Size: 115.6 kB
Tags: Python 3
Uploaded using Trusted Publishing? No
Uploaded via: twine/6.2.0 CPython/3.12.10

File hashes

Hashes for intelliscrape-2.5.0-py3-none-any.whl
Algorithm	Hash digest
SHA256	`e310e385e113836abf73663527b63938f7d21dc92359e48b5e9fb316f83ced9e`
MD5	`a5c73c7609b73f78319a7ec58d83797e`
BLAKE2b-256	`24710ae695914ba3dc121b444c2216e793d1005a518d9cbc020280e54120ae90`

See more details on using hashes here.

Release history Release notifications | RSS feed

Aug 19, 2026 2 files

Aug 3, 2026 2 files

Aug 2, 2026 2 files

Aug 2, 2026 2 files

Aug 2, 2026 2 files

Jul 31, 2026 2 files

Jul 31, 2026 2 files

Jul 31, 2026 2 files

Jul 30, 2026 2 files

Jul 30, 2026 2 files

This release

2.5.0 This release

Jul 29, 2026 2 files

Jul 29, 2026 2 files

Jul 23, 2026 2 files

Mar 6, 2026 2 files

Feb 26, 2026 2 files

intelliscrape 2.5.0

IntelliScrape

Table of Contents

Installation

Optional Extras

System Requirements

Quick Start

CLI

Python — One-liner

Python — Full-featured class

Get structured data

Crawl entire website

CLI Reference

Output Options

Intelligent Mode

Engine Control

Proxy Options

Authentication

Cookies

Request Modification

Pagination & Search

Crawl

Downloads

Export

CLI Examples

Library API Reference

scrape() — Quick One-liner

IntelliScrape — Main Class

Constructor Parameters

Methods

scrape(url, **kwargs)

get_structured(url, **kwargs)

analyze(url)

scrape_many(urls, **kwargs)

check_captcha(url)

check_antibot(url)

find_free_proxies(test=True)

get_proxy_status()

crawl() — Website Crawler

AsyncIntelliScrape — Async Scraping

DataExporter — Export Formats

Downloader — File Downloads

Authenticator — Login & Sessions

FormSubmitter — Form Interaction

Paginator — Auto-pagination

RequestInterceptor — Request/Response Modification

CookieManager — Cookie Persistence

CaptchaDetector & CaptchaSolver — CAPTCHA Handling

AntiBotDetector — Anti-bot Vendor Detection

Anti-bot Bypass Classes

Engine System — 4-Tier Escalation

Intelligent Mode

Anti-Detection

CAPTCHA Solving

Automated (via API)

Manual CAPTCHA Solving

Proxy Configuration

Single Proxy

Multiple Proxies

Residential Proxy Providers

Free Proxies (Automatic)

Export Formats

CLI

Python

Async Support

Advanced Usage

Scrape with Login

Scrape with Custom Headers

Block URLs

Scrape React/Vue/Angular SPAs

Persistent Sessions

Download Files

Troubleshooting

Site returns empty or accessibility widget text

CAPTCHA blocking scraping

Getting blocked by anti-bot

Playwright not installed

Camoufox not installed

nodriver not installed

Project Structure

`scrape()` — Quick One-liner

`IntelliScrape` — Main Class

`scrape(url, **kwargs)`

`get_structured(url, **kwargs)`

`analyze(url)`

`scrape_many(urls, **kwargs)`

`check_captcha(url)`

`check_antibot(url)`

`find_free_proxies(test=True)`

`get_proxy_status()`

`crawl()` — Website Crawler

`AsyncIntelliScrape` — Async Scraping

`DataExporter` — Export Formats

`Downloader` — File Downloads

`Authenticator` — Login & Sessions

`FormSubmitter` — Form Interaction

`Paginator` — Auto-pagination

`RequestInterceptor` — Request/Response Modification

`CookieManager` — Cookie Persistence

`CaptchaDetector` & `CaptchaSolver` — CAPTCHA Handling

`AntiBotDetector` — Anti-bot Vendor Detection