lazy-crawler 2.0.0

A powerful and extensible web crawling framework built with Scrapy and Playwright.

Lazy Crawler

Extensible web crawling and data extraction framework.

A technical foundation for building scalable data pipelines using Scrapy and Playwright.

Core Stack
Code Quality
Documentation

---

Lazy Crawler is an extensible web crawling framework designed for developers who need to build high-performance data extraction pipelines. It combines the speed of Scrapy with the dynamic rendering capabilities of Playwright to handle complex web environments.

Features

• Automated Workflows: Streamlined project structure for rapid deployment of new spiders.
• Dynamic Content Support: Built-in Playwright integration for rendering JavaScript-heavy applications.
• Multi-backend Support: Native export functionality for MongoDB, PostgreSQL, CSV, JSON, and Google Sheets.
• Developer First: Focuses on clean abstractions and extensibility over rigid configurations.
• Resilient Extraction: Integrated support for proxy rotation and anti-detection measures.

Quick Start

1. Installation

This project uses uv for dependency management.

uv pip install .

For development:

# Initialize and install in editable mode
uv pip install -e .

[!NOTE] Install Playwright browser binaries after the initial setup: playwright install

2. Static Site Crawler

Create my_agent.py:

import scrapy
from lazy_crawler.crawler.spiders.base_crawler import LazyBaseCrawler
from scrapy.crawler import CrawlerProcess

class MyAgent(LazyBaseCrawler):
    name = "my_agent"

    def start_requests(self):
        yield scrapy.Request("https://example.com", self.parse)

    def parse(self, response):
        yield {
            "title": response.css("h1::text").get(),
            "url": response.url
        }

process = CrawlerProcess()
process.crawl(MyAgent)
process.start()

3. Dynamic Content (JavaScript)

Leverage Playwright for sites that require browser rendering:

class DynamicAgent(LazyBaseCrawler):
    name = "dynamic"

    def start_requests(self):
        yield scrapy.Request(
            "https://dynamic-site.com",
            meta={"playwright": True},
            callback=self.parse
        )

    def parse(self, response):
        data = response.css(".rendered-content::text").get()
        yield {"content": data}

Data Management

1. MongoDB Integration

Configuration (.env):

MONGO_URI=mongodb://localhost:27017
MONGO_DATABASE=lazy_crawler_db

Settings:

ITEM_PIPELINES = {
    "lazy_crawler.crawler.pipelines.MongoPipeline": 400,
}

2. Google Sheets Export

Configuration (.env):

GOOGLE_SHEETS_CREDS_FILE=creds.json
GOOGLE_SHEETS_SPREADSHEET_NAME=CrawlData
GOOGLE_SHEETS_WORKSHEET_NAME=Results

Dashboard & API

The project includes a dashboard for monitoring crawl progress and exploring extracted data.

Start the service:

uv run python -m lazy_crawler.app.main

• Dashboard: http://localhost:8000/
• API Documentation: http://localhost:8000/docs

Docker Deployment (Production)

Deploy using the provided orchestration files:

# Quick deployment
./deploy.sh

# Manual startup
docker compose up --build -d

• Dashboard: http://localhost/
• API Docs: http://localhost/docs
• Health: http://localhost/health

Customization

The framework is designed to be modified. You can extend LazyBaseCrawler or implement custom pipelines to handle specific data requirements.

Contributing

Technical contributions and bug reports are welcome. Please refer to CONTRIBUTING.md.

License

Lazy Crawler is licensed under the MIT License.

---

Created by Pradip P.