qrawlix 1.0.0

High-performance async web scraping and asset retrieval with TLS fingerprint emulation and concurrent processing.

""" Qrawlix — High-Performance Asynchronous Web Scraping & Asset Retrieval

Qrawlix is a research-oriented Python library for fetching web pages, extracting structured data, and downloading binary assets (images, documents, archives) at scale. It is built on top of curl_cffi, which provides browser-grade TLS fingerprint emulation — allowing requests to appear as though they originate from standard browsers (Chrome, Firefox, Safari, Edge).

Key capabilities

• TLS fingerprint rotation — automatic per-request browser identity cycling to improve compatibility with diverse server configurations.
• Concurrent scraping — fetch dozens of pages simultaneously with configurable parallelism and built-in rate-limiting.
• Concurrent asset downloading — download hundreds of images or files in parallel, with per-directory organisation.
• Structured data extraction — declarative CSS-selector-based rules for pulling titles, prices, links, and other fields from HTML.
• Multi-strategy request routing — automatic fallback through secondary channels when a direct connection is unavailable.
• Parallel engine — high-level orchestrator that handles the full pipeline (scrape → extract → download → organise) in one call.

Installation

::

pip install qrawlix

Or from source::

git clone https://github.com/YOUR_USER/qrawlix.git
cd qrawlix
pip install .

Quick start

Scrape a single page and extract the title::

import asyncio
from qrawlix import QrawlixClient

async def main():
    client = QrawlixClient()
    result = await client.scrape("https://example.com")
    if result["success"]:
        title = result["parser"].text("h1")
        print(f"Page title: {title}")

asyncio.run(main())

Scrape multiple pages and download all discovered images in one call::

from qrawlix.parallel import scrape_pages

results = asyncio.run(scrape_pages([
    "https://site-a.com/gallery",
    "https://site-b.com/chapter/42",
]))

print(f"{results['succeeded']}/{results['targets']} sites processed")
print(f"{results['images_downloaded']} images downloaded")

Output is automatically organised::

qrawlix_output/
├── summary.json
├── site-a.com/
│   ├── page.html
│   ├── data.json
│   └── images/
│       ├── photo_001.jpg
│       └── photo_002.png
└── site-b.com/
    ├── page.html
    ├── data.json
    └── images/
        └── ...

Core API

QrawlixClient — async HTTP client

::

    from qrawlix import QrawlixClient

    client = QrawlixClient(
        max_concurrency=10,   # max simultaneous requests
        timeout=15.0,         # per-request timeout (seconds)
        retries=1,            # retry attempts on failure
    )

    # Single URL
    result = await client.scrape("https://example.com")
    # result["success"]         — bool
    # result["html"]            — raw response text
    # result["parser"]          — ContentParser instance
    # result["elapsed_ms"]      — request latency
    # result["status_code"]     — HTTP status

    # Structured extraction
    result = await client.scrape("https://example.com", rules={
        "title": {"selector": "h1", "type": "text"},
        "links": {"selector": "a", "type": "attr_all", "attr": "href"},
    })
    # result["extracted_data"]  — dict with extracted fields

    # Multiple URLs concurrently
    batch = await client.scrape_many([
        "https://site-a.com",
        "https://site-b.com",
    ])
    # batch["successful"]       — count of successful fetches
    # batch["results"]          — list of per-URL result dicts


**ContentParser** — HTML / markdown parser

::

parser = result["parser"]          # obtained from QrawlixClient.scrape()

parser.text("h1")                  # first match inner text
parser.text_all("p")               # all match inner texts
parser.attribute("a", "href")      # first match attribute
parser.attribute_all("img", "src") # all match attributes

# Declarative extraction
data = parser.extract_by_rules({
    "title":   {"selector": "h1",              "type": "text"},
    "price":   {"selector": ".price",          "type": "text"},
    "image":   {"selector": "img.product",     "type": "attr",  "attr": "src"},
    "gallery": {"selector": ".gallery img",    "type": "attr_all", "attr": "src"},
})

# List extraction (repeating containers)
items = parser.extract_list("div.product-card", {
    "name":  {"selector": "h2",   "type": "text"},
    "link":  {"selector": "a",    "type": "attr", "attr": "href"},
})

AssetPipeline — concurrent binary downloader

::

    from qrawlix import AssetPipeline

    pipeline = AssetPipeline(
        output_dir="downloads",
        concurrency=20,          # max simultaneous downloads
    )
    dl = await pipeline.download_all([
        "https://example.com/photo1.jpg",
        "https://example.com/photo2.png",
    ])
    # dl["total_downloaded"]   — success count
    # dl["total_bytes"]        — total size downloaded
    # dl["results"]            — per-asset status dicts


**ParallelEngine** — full pipeline orchestrator
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

::

    from qrawlix import ParallelEngine

    engine = ParallelEngine(
        scrape_concurrency=8,    # concurrent page fetches
        download_concurrency=24, # concurrent asset downloads
    )
    summary = await engine.run([
        "https://site-a.com/gallery",
        "https://site-b.com/chapter/1",
    ], output_root="my_output")

    # One-liner equivalent:
    from qrawlix.parallel import scrape_pages
    summary = await scrape_pages(urls, output="my_output")


Configuration
-------------
Concurrency levels are automatically derived from system CPU count
when not explicitly set.  The default heuristic is ``max(4, min(cpu*2, 32))``.

Custom image selectors can be passed to the parallel engine for
domain-specific extraction::

    engine = ParallelEngine(image_selectors={
        "my-site.com": [".custom-gallery img", "img.content"],
    })

Important notes
---------------
- This library is provided for **research and educational purposes**.
  Users are responsible for complying with the terms of service of any
  website they interact with.
- Qrawlix uses public content-reader proxies and translation services
  as secondary request channels when a direct connection is unavailable.
  These are best-effort fallbacks and may not work for all endpoints.
- The library does **not** execute JavaScript.  Pages that rely entirely
  on client-side rendering (e.g., Google Images, React SPAs) will return
  limited content from the initial HTML payload.

License
-------
MIT License — see the LICENSE file for details.
"""