A “soft” 404 page is a page that is served with 200 status, but is really a page that says that content is not available.
Installation
pip install soft404
Usage
The easiest way is to use the soft404.probability function:
>>> import soft404
>>> soft404.probability('<h1>Page not found</h1>')
0.9736860086882132
You can also create a classifier explicitly:
>>> from soft404 import Soft404Classifier
>>> clf = Soft404Classifier()
>>> clf.predict('<h1>Page not found</h1>')
0.9736860086882132
Development
Classifier is trained on 120k pages from 25k domains, with 404 page ratio of about 1/3. With 10-fold cross-validation, PR AUC (average precision) is 0.990 ± 0.003, and ROC AUC is 0.995 ± 0.002.
Getting data for training
Install dev requirements:
pip install -r requirements_dev.txt
Run the crawler for a while (results will appear in pages.jl.gz file):
cd crawler scrapy crawl spider -o gzip:pages.jl -s JOBDIR=job
Training
First, extract text and structure from html:
./soft404/convert_to_text.py pages.jl.gz items
This will produce two files, items.meta.jl.gz and items.items.jl.gz. Next, train the classifier:
./soft404/train.py items
Vectorizer takes a while to run, but it’s result is cached (the filename where it is cached will be printed on the next run). If you are happy with results, save the classifier:
./soft404/train.py items --save soft404/clf.joblib
License
License is MIT.
Release files for soft404 0.2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| soft404-0.2.1.tar.gz | 30.3 kB | Details |
Release files / soft404-0.2.1.tar.gz
| Download URL | soft404-0.2.1.tar.gz |
|---|---|
| Size | 30.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
09d2cf8bd6264d542f3f8613e3d8693e229eca149b203b30822506e4e0805c4f
|
|
BLAKE2b-256 checksum How to use checksums |
de161691f87f56a6a8ef0bbd164b389b1ab97a3fe82556921a1d619dcb5ba6a7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
