qwlcrapstar 0.1.11

Universal AI-Powered Web Scraper Library

QwlCrapstar: Autonomous AI Data Extraction Agent

QwlCrapstar is a high-performance, agentic web scraping library designed for professional data extraction. It leverages Large Language Models (LLMs) to transform unstructured web content into validated, structured data without the need for brittle CSS selectors or XPaths.

The Core Concept (The "What")

Traditional web scraping relies on identifying specific HTML tags and classes. When a website's layout changes, the scraper breaks. QwlCrapstar operates on Semantic Intent. By providing a natural language "Mission" and a target data structure (Schema), the library uses AI to understand the context of the page, navigate its structure, and extract exactly what is requested, regardless of underlying code changes.

Why QwlCrapstar? (The "Why")

1. Resilience: Scrapers do not break when a website shifts from a "table" layout to a "flexbox" layout.
2. Reduced Development Time: No need to spend hours inspecting DOM structures. If a human can read the data, QwlCrapstar can extract it.
3. Complex Logic Handling: Built-in support for semantic filtering and data normalization (e.g., converting "2 hours ago" into a standardized ISO date).
4. Local & Cloud Flexibility: Support for elite cloud models (Perplexity, OpenAI) and private local models (Ollama).

---

Installation

pip install qwlcrapstar
playwright install chromium

---

Getting Started (The "How")

Level 1: Basic Mission

The most direct way to use the library is to provide a URL and a natural language instruction. QwlCrapstar will automatically infer the appropriate schema and execute the mission.

import asyncio
from qwl_crapstar import QwlCrapstar

async def main():
    # Automatically detects API keys in environment variables
    scraper = QwlCrapstar() 
    
    # Mission: Extract the latest 10 technology news items
    results = await scraper.scrape(
        url="https://news.ycombinator.com",
        prompt="Find 10 interesting technology stories"
    )
    print(results)

asyncio.run(main())

---

Professional Developer Guide

1. Advanced Schema Definition

For production applications, explicitly defining your data structure is recommended for consistency. QwlCrapstar supports class-based schemas using a specialized Field system.

from qwl_crapstar import QwlCrapstar, Schema, Field

class ProductSchema(Schema):
    name = Field("Exact product name", required=True)
    price = Field("Selling price", type=float, hint="Look for currency symbols")
    description = Field("Short item summary")
    specs = Field("Technical specifications", type=dict)

scraper = QwlCrapstar()
results = await scraper.scrape(
    url="https://example-shop.com/product/1",
    schema=ProductSchema
)

2. The Semantic Logic Engine

You can enforce strict extraction constraints using programmatic rules. This acts as a filter on the LLM's interpretation.

results = await scraper.scrape(
    url="https://linkedin.com/jobs",
    prompt="Senior Data Scientist roles",
    rules=[
        "Filter: Only include postings from the last 7 days",
        "Normalize: All salaries must be expressed as annual USD",
        "Constraint: Employer must be a Fintech company"
    ]
)

3. Complexity & Resource Management

The complexity_level parameter controls the depth of analysis and the resources allocated to the mission.

Level	Use Case	Rationale
basic	Simple static pages	Minimal latency, lower token consumption.
standard	Most content websites	Balanced rendering wait and context window.
advanced	Dynamic JS-heavy sites	Increased rendering wait and deeper context analysis.
elite	Critical/Protected apps	Maximum token window and rigorous structural analysis.

4. Enterprise Processing Pipeline

Post-extraction, you can route data through a series of processors for cleaning and deduplication.

from qwl_crapstar.core.processors import Pipeline, DataValidator, Deduplicator

# Build the pipeline
pipeline = Pipeline([
    DataValidator(schema=ProductSchema),
    Deduplicator(fields=["name", "price"])
])

# Process raw results
sanitized_data = await pipeline.run(raw_results)

---

Configuration & Credentials

QwlCrapstar features Auto-Discovery. It scans the following environment variables automatically: OPENAI_API_KEY, ANTHROPIC_API_KEY, GROQ_API_KEY, DEEPSEEK_API_KEY, PERPLEXITY_API_KEY.

LLM Provider	Implementation Note
Perplexity	Best for real-time web content and citation-accuracy.
OpenAI	Industry standard for high-fidelity structured output.
Groq	Exceptional speed for high-volume scraping tasks.
Ollama	Entirely local; ensures zero data leaves your infrastructure.

---

Technical Architecture

The library operates through a three-stage pipeline:

1. Browser Execution: A Playwright-based engine navigates to the target, handles rendering, and extracts a sanitized version of the DOM.
2. Semantic Mapping: The LLM analyzes the content window based on the provided Mission, Schema, and Complexity level.
3. Serialization: The LLM returns a validated JSON object that matches the requested schema constraints.

License

MIT License.