scrapy-beautifulsoup
Simple Scrapy middleware to process non-well-formed HTML with BeautifulSoup
Installation
The package is on PyPI and can be installed with pip:
pip install scrapy-beautifulsoup
Configuration
Add the middleware to DOWNLOADER_MIDDLEWARES dictionary setting:
DOWNLOADER_MIDDLEWARES = {
'scrapy_beautifulsoup.middleware.BeautifulSoupMiddleware': 400
}
By default, BeautifulSoup would use the built-in html.parser parser. To change it, set the BEAUTIFULSOUP_PARSER setting:
BEAUTIFULSOUP_PARSER = "html5lib" # or BEAUTIFULSOUP_PARSER = "lxml"
html5lib is an extremely lenient parser and, if the target HTML is seriously broken, you might consider being it your first choice. Note: html5lib has to be installed in this case:
pip install html5lib
Motivation
BeautifulSoup itself with the help of an underlying parser of choice does a pretty good job of handling non-well-formed or broken HTML. In some cases, it makes sense to pipe the HTML through BeautifulSoup to “fix” it.
1 maintainer
Release files for scrapy-beautifulsoup 0.0.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| scrapy-beautifulsoup-0.0.2.tar.gz | 2.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| scrapy_beautifulsoup-0.0.2-py2.py3-none-any.whl | Python 3, Python 2 | none | any | Details |
Total release size:6.9 kB
Release files / scrapy-beautifulsoup-0.0.2.tar.gz
| Download URL | scrapy-beautifulsoup-0.0.2.tar.gz |
|---|---|
| Size | 2.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
6cf3158d257bb3d95dc45b8892d35dbf1d356afa4c33d4b1829fb34cdfbbd3be
|
|
BLAKE2b-256 checksum How to use checksums |
83533b51bc3dc26e4007241f9bcdb9693501e026192898be39cfe33791db0fff
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Release files / scrapy_beautifulsoup-0.0.2-py2.py3-none-any.whl
| Download URL | scrapy_beautifulsoup-0.0.2-py2.py3-none-any.whl |
|---|---|
| Size | 4.5 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
354fb34f6d302768cb2e6380464c3310934af1de673714f3d6c46b8d0f88c3a1
|
|
BLAKE2b-256 checksum How to use checksums |
70067c0f6a2f0a595cfa767fb123635f1b347fc23fe60d5f3b94eabc19582520
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
