soft-404: a classifier for detecting soft 404 pages
A “soft” 404 page is a page that is served with 200 status, but is really a page that says that content is not available.
Installation
pip install soft-404
Usage
The easiest way is to use the soft404.probability function:
>>> import soft404
>>> soft404.probability('<h1>Page not found</h1>')
0.9736860086882132
You can also create a classifier explicitly:
>>> from soft404 import Soft404Classifier
>>> clf = Soft404Classifier()
>>> clf.predict('<h1>Page not found</h1>')
0.9736860086882132
Development
Classifier is trained on 198801 pages from 35995 domains, with 404 page ratio of about 1/3. With 10-fold cross-validation, PR AUC (average precision) is 0.991 ± 0.002, and ROC AUC is 0.995 ± 0.002.
Getting data for training
Install dev requirements:
pip install -r requirements_dev.txt
Run the crawler for a while (results will appear in pages.jl.gz file):
cd crawler scrapy crawl spider -o pages.jl.gz -t jl.gz -s JOBDIR=job
Training
First, extract text and structure from html:
./soft404/convert_to_text.py pages.jl.gz items
This will produce two files, items.meta.jl.gz and items.items.jl.gz. Next, train the classifier:
./soft404/train.py items
Vectorizer takes a while to run, but it’s result is cached (the filename where it is cached will be printed on the next run). If you are happy with results, save the classifier:
./soft404/train.py items --save soft404/clf.joblib
License
License is MIT.
1 maintainer
Release files for soft-404 0.4.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| soft-404-0.4.0.tar.gz | 31.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| soft_404-0.4.0-py2.py3-none-any.whl | Python 2, Python 3 | none | any | Details |
Total release size: 62.2 kB
Release files / soft-404-0.4.0.tar.gz
| Download URL | soft-404-0.4.0.tar.gz |
|---|---|
| Size | 31.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1a549ae53e97f1b17718d54c96ce4b84baa8ac00f23ab7a78d177895f3fb4d30
|
|
BLAKE2b-256 checksum How to use checksums |
5c0e3c313d29f2f642f5f6afa75b54960d0f7ff7d75e714efbf17e827afdbf8a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/50.1.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.6.7
|
Release files / soft_404-0.4.0-py2.py3-none-any.whl
| Download URL | soft_404-0.4.0-py2.py3-none-any.whl |
|---|---|
| Size | 30.9 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
e89384c161f7fcbbb3a5bba5aae7d409335b157e4d69b3d554244e51b767f6be
|
|
BLAKE2b-256 checksum How to use checksums |
bdc61960aa0dfe3875e69e07a62401cd459ca414d8595421f8bf4cc1696ffd93
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/50.1.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.6.7
|
