soft404: a classifier for detecting soft 404 pages
A “soft” 404 page is a page that is served with 200 status, but is really a page that says that content is not available.
Installation
pip install soft404
Usage
The easiest way is to use the soft404.probability function:
>>> import soft404
>>> soft404.probability('<h1>Page not found</h1>')
0.9736860086882132
You can also create a classifier explicitly:
>>> from soft404 import Soft404Classifier
>>> clf = Soft404Classifier()
>>> clf.predict('<h1>Page not found</h1>')
0.9736860086882132
Development
Classifier is trained on 120k pages from 25k domains, with 404 page ratio of about 1/3. With 10-fold cross-validation, PR AUC (average precision) is 0.990 ± 0.003, and ROC AUC is 0.995 ± 0.002.
Getting data for training
Install dev requirements:
pip install -r requirements_dev.txt
Run the crawler for a while (results will appear in pages.jl.gz file):
cd crawler scrapy crawl spider -o gzip:pages.jl -s JOBDIR=job
Training
First, extract text and structure from html:
./soft404/convert_to_text.py pages.jl.gz items
This will produce two files, items.meta.jl.gz and items.items.jl.gz. Next, train the classifier:
./soft404/train.py items
Vectorizer takes a while to run, but it’s result is cached (the filename where it is cached will be printed on the next run). If you are happy with results, save the classifier:
./soft404/train.py items --save soft404/clf.joblib
License
License is MIT.
Release files for soft404 0.2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| soft404-0.2.1.tar.gz | 30.3 kB | Without provenance | Details for soft404-0.2.1.tar.gz |
Release files / soft404-0.2.1.tar.gz
| Download URL | soft404-0.2.1.tar.gz |
|---|---|
| Size | 30.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
09d2cf8bd6264d542f3f8613e3d8693e229eca149b203b30822506e4e0805c4f
|
|
BLAKE2b-256 checksum How to use checksums |
de161691f87f56a6a8ef0bbd164b389b1ab97a3fe82556921a1d619dcb5ba6a7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Security for soft404 0.2.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
