scrapework 0.4.6

simple scraping framework

Scrapework

Scrapework is a simple and opiniatated framework to extract data from the web. It is inspired by Scrapy and designed for simple tasks and easy management, allowing you to focus on the scraping logic. It is built on top of parsel (used by Scrapy) and httpx libraries. Some of the key differences are:

• No CLI
• No twisted framework
• Designed for in-process usage

Installation

First, clone the repository or install as a dependencies:

With pip:

pip install scrapework

With poetry:

poetry add scrapework

Quick Start

Create a Scraper

First, create a Scraper class to define how extract data and optionally navigate a website. Here's how you can create a simple Scraper:

from scrapework.scraper import Scraper

class SimpleScraper(Scraper):
    name = "simple_scraper"

    def extract(self, ctx, selector):
        for quote in selector.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('span small::text').get(),
            }

    def process(self, items, config):
        for item in items:
            print(f"Quote: {item['text']}, Author: {item['author']}")

Similar to Scrapy parse, the extract method is an expected and this is where you define your scraping logic. It's called with the HTTP response of the initial URL.

Run the Scraper

To run the Scraper, you need to create an instance and call the run method passing the URLs to scrape:

scraper = SimpleScraper()
scraper.run(['http://quotes.toscrape.com'])

Modules Configuration

Scrapework can be extended using various modules:

• middleware to configure the request handling (chache, proxy, ...).
• handlers: comes with various handlers (log, save to file, save to database.)
• reporters: to export and log the scraping events and metadata.

Flow

The scraping flow consists of the following steps:

1. Webpage downloading: Fetch the webpages using httpx. Optionally, use middleware to handle requests.
2. Extract data: Extract structured data from the HTML using parsers.
3. Export data: Use handlers to store or export the structured data.
4. Reporting: Generate reports and logs of the scraping process using reporters.

For more details see Design.

Advanced Usage

For more advanced usage, you can override other methods in the Scraper, Parser, and Pipeline classes. Check the source code for more details.

Add Parser

Alternatively, you can create a Parser class to define how to extract data from a webpage. Here's how you can create an Parser and configure it in the Scraper:

from scrapework.parsers import Parser, Scraper

class SimpleScraper(Scraper):
    name = "simple_scraper"
    parser = SimpleParser()

class SimpleParser(Parser):
    def extract(self, ctx, selector):
        return {
            'text': selector.css('span.text::text').get(),
            'author': selector.css('span small::text').get(),
        }

The extract method is where you define your extraction logic. It's called passing a parsel.Selector object that you can use to extract data from the HTML using css or xpath.

Add a data handler

Similar to a pipeline, an handler defines how to process and store the data:

from scrapework.handlers import Handler

class SimpleHandler(Handler):
    def process_items(self, items, config):
        for item in items:
            print(f"Quote: {item['text']}, Author: {item['author']}")

The process_items method is where you define your processing logic. It's called with the items extracted by the Parser and a PipelineConfig object.

scraper = SimpleScraper()

scraper.use(SimpleHandler())

Testing

To run the tests, use the following command:

pytest tests/

Contributing

Contributions are welcome! Please read the contributing guidelines first.

License

Scrapework is licensed under the MIT License.