scrapy-h5 0.1.1

Scrapy HTML5 support

Parsel H5

Scrapy integration for the html5ever and lexbor HTML parsers.

This package provides a Scrapy Downloader Middleware that replaces the default lxml-based HTML parsing with a HTML5 one.

Why html5ever?

• Better HTML5 compliance: Parses HTML the way browsers do
• Handles malformed HTML gracefully: More forgiving with real-world HTML
• As fast as Parsel: Rust-based parser with Python bindings (markupever)

Why Lexbor?

• Fastest HTML5 parser: C-based parser with Python bindings (selectolax)
• Better HTML5 compliance: Parses HTML the way browsers do
• Handles malformed HTML gracefully: More forgiving with real-world HTML

Installation

pip install scrapy-h5

Or with uv:

uv add scrapy-h5

Quick start

1. Enable the middleware in your Scrapy project

Add to your settings.py:

DOWNLOADER_MIDDLEWARES = {
    'scrapy_h5.HtmlFiveResponseMiddleware': 950,
}

# Optional: disable globally (backend by default)
# HTML5_BACKEND = False

2. Use in your spider

import scrapy


class MySpider(scrapy.Spider):
    name = 'myspider'
    start_urls = ['https://example.com']

    def parse(self, response):
        # CSS selectors work as expected
        titles = response.css('h1::text').getall()

        # Attribute extraction
        links = response.css('a::attr(href)').getall()

        # Chained selectors
        for item in response.css('div.product'):
            yield {
                'name': item.css('h2::text').get(),
                'price': item.css('.price::text').get(),
                'url': item.css('a::attr(href)').get(),
            }

XPath and JMESPath support

XPath and JMESPath selectors are not supported. Use CSS selectors instead.

Per-request control

You can change/disable html5 backend per request using meta:

def start_requests(self):
    # HTML5 parsing backend (default)
    yield scrapy.Request(url, callback=self.parse)

    # Disable html5 for this request (use lxml instead)
    yield scrapy.Request(
        url2,
        callback=self.parse_legacy,
        meta={'use_html5': False}
    )


def parse_with_html5(self, response):
    # Force html5 even if HTML5_BACKEND=False
    yield scrapy.Request(
        url,
        callback=self.parse,
        meta={'use_html5': 'html5ever'}
    )

API reference

Classes

• HtmlFiveSelector: Selector class wrapping html5ever and lexbor elements
• HtmlFiveSelectorList: List of selectors with bulk operations
• HtmlFiveResponse: Response class with html5-based selector
• HtmlFiveResponseMiddleware: Scrapy Downloader Middleware that replaces HtmlResponse with HtmlFiveResponse

Exceptions

• XPathConversionError: Raised when an XPath expression cannot be converted to CSS
• HtmlFiveParseError: Raised when HTML parsing fails
• HtmlFiveSelectorError: Base exception for selector errors
• HtmlFiveSelectError: Raised when CSS selection fails

Settings

Setting	Default	Description
HTML5_BACKEND	lexbor	Global html5 backend. lexbor and html5ever enables, False disables

Request meta

Key	Type	Description
use_html5	bool	Per-request override. lexbor and html5ever enables, False disables

Middleware priority

The default priority (950) is chosen to run:

• After HttpCompressionMiddleware (590) - responses are decompressed
• After HttpCacheMiddleware (900) - cached responses are handled
• Before most other processing

Adjust the priority if needed:

DOWNLOADER_MIDDLEWARES = {
    'scrapy_h5.HtmlFiveResponseMiddleware': 400,  # Earlier in the chain
}

License

MIT