Turn a web crawl into an ML-ready dataset: extract, filter, deduplicate, embed, shard.
Verified details
These details have been verified by PyPIProject links
GitHub Statistics
Maintainers
Project description
websieve
Turn a web crawl into an ML-ready dataset. Extract, filter, deduplicate, embed, shard.
What this actually is
You scraped 50,000 pages. You cannot train on them, because a scraped page is not a document.
In: raw HTML, one JSON object per line.
{"url": "https://example.com/gpu-guide",
"html": "<html><head><title>GPU scheduling on Kubernetes</title></head><body><nav><a href='/'>Home</a><a href='/about'>About</a></nav><article><h1>GPU scheduling</h1><p>Kubernetes exposes GPUs through the NVIDIA device plugin, which advertises nvidia.com/gpu as an allocatable resource on every node...</p></article><footer>© 2026 Example Inc. Privacy Policy</footer></body></html>"}
Out: clean text, deduplicated, quality-scored, in gzipped shards a dataloader can stream.
{"doc_id": "dc1bf1ffa1945c31",
"url": "https://example.com/gpu-guide",
"title": "GPU scheduling on Kubernetes",
"text": "GPU scheduling\n\nKubernetes exposes GPUs through the NVIDIA device plugin, which advertises...",
"quality": {"passed": true, "rules": {"word_count": {"passed": true, "value": 104, "threshold": 50}}},
"signatures": {"raw": "81879111...", "normalized": "0745f9c2...", "structural": "a74bccb9..."}}
The nav, the footer, and the copyright line are gone. The title came from <title>, the body from
the article, and nothing else survived. Across a real crawl, so do the 9,884 pages that were the
same article under a different URL and the 14,022 that were link farms, stub pages, or SEO spam.
Both blocks above are actual output, not illustration. docs/quickstart.md reproduces them in
about a minute.
That gap is the whole product. Crawling is solved; Scrapy does it. Training is solved; your framework does it. The part in between, where a crawl becomes a corpus, is where people quietly lose weeks and then train on duplicates anyway.
Who this is for
- You are building an LLM or RAG corpus from crawled pages and need dedup that catches near duplicates, not just byte-identical ones.
- You run a scraper in production and want the cleaning stage to be a library with tests rather
than a 400-line
clean.pynobody wants to touch. - You cannot install dependencies where the cleaning has to run: someone else's container, a locked-down build box, an air-gapped environment.
If you want a distributed, Spark-scale pipeline with a cluster behind it, use HuggingFace's
datatrove. websieve deliberately runs in one process with no dependencies, which is the right
trade below roughly ten million documents and the wrong one above it.
How it works
crawl -> extract -> normalize -> exact dedup -> quality -> near dedup -> embed -> shards
boilerplate NFKC 3 levels 9 rules MinHash+LSH batched
Each stage costs more than the one before it, so each shrinks the input to the next. One hash beats
nine heuristics; nine heuristics beat 128 hash permutations. Details in
docs/architecture.md.
The core has no dependencies. Not "few". None. pyarrow, torch, and scrapy are optional
extras used only by the stages that genuinely need them, and CI fails if a third-party import ever
reaches the core.
Install
pip install websieve # core, zero dependencies
pip install "websieve[parquet]" # + Parquet output
pip install "websieve[embed]" # + GPU embedding
pip install "websieve[all]"
Use
Pipe a crawl straight in:
scrapy crawl myspider -o - -t jsonlines | websieve build - -o dataset/
Or run against a file:
websieve build crawl.jsonl -o dataset/ --threshold 0.85 --shard-size 50000
You get sharded output, a manifest, and a stats report:
dataset/
shard-00000.jsonl.gz
shard-00001.jsonl.gz
manifest.json every shard, its record count, its size
stats.json what was dropped, and by which rule
seen 50000
kept 18342 (36.7%)
dropped 31658
empty 412
exact duplicate 9884
quality 14022
near duplicate 7340
quality rule failures (a document can fail several):
word_count 8110
terminal_punctuation_ratio 6033
repetition_ratio 2914
That breakdown is the point. A filter you cannot attribute is a filter you cannot tune.
Inspect before you commit
websieve assess crawl.jsonl -v # what would be dropped, and why. Drops nothing.
websieve assess crawl.jsonl --sample 1000 # same, on a sample, for large crawls
websieve dedup crawl.jsonl # duplicate clusters with similarity scores
websieve extract page.html # main-content text from one page
As a library
from websieve.pipeline import Pipeline, PipelineConfig
from websieve.models import Document
pipeline = Pipeline(PipelineConfig(near_dup_threshold=0.85))
for doc in pipeline.process(Document(url=u, html=h) for u, h in crawl()):
index(doc.text)
print(pipeline.stats.render())
Every stage also stands alone:
from websieve.quality.heuristics import assess
from websieve.dedup.minhash import MinHash, LSHIndex
from websieve.clean.boilerplate import extract
What each stage does
Extraction
Text-density heuristic, not a readability port. Blocks are scored by length and link density, then the best contiguous run is kept, so navigation and footers fall away because they are short and mostly links. Headings adjacent to the body are reclaimed, because Kadane's algorithm will not pick them up on its own and an article without its title is a worse document.
No dependencies. If you can afford one and need higher accuracy, use trafilatura and feed its
output into the quality stage instead.
Quality
Nine rules from the published recipes for large web corpora, chiefly the Gopher rules (Rae et al., 2021) and the C4 cleanup (Raffel et al., 2020): word count, mean word length, symbol ratio, alphabetic ratio, bullet and ellipsis line ratios, terminal punctuation, line repetition, and boilerplate markers.
Reimplemented rather than vendored so every threshold is visible and adjustable. The right values genuinely differ between general web text and a domain corpus, and you cannot tune what you cannot see. Every rule runs even after one fails, so the failure histogram is complete.
Deduplication
Two passes, cheap before expensive.
Exact, at three levels. raw is byte-identical. normalized ignores case, punctuation, and
spacing. structural also collapses digits, which catches templated pages differing only by a
price, date, or id. That last one will merge genuinely different pages whose only distinguishing
content is numeric, so choose it deliberately.
Near, by MinHash with LSH banding. Shingle into word n-grams, keep the minimum under num_perm
hash permutations, then band the signature so candidate lookup is a hash hit instead of an O(n^2)
scan. Candidates are verified against real signature similarity afterwards, because LSH returns
candidates, not answers.
Permutations are simulated as (a*h + b) mod p over a Mersenne prime, the standard universal
hashing construction, which is why this needs no numpy. Accuracy is what the theory predicts:
| Pair | True Jaccard | MinHash estimate, 256 perms |
|---|---|---|
| One word differs | 0.833 | 0.863 |
| Unrelated documents | 0.000 | 0.000 |
| Identical | 1.000 | 1.000 |
Embedding
The model call sits behind a Protocol, so the part that actually governs throughput is testable
without a GPU and CI exercises it with a stub.
Two things dominate embedding throughput, and neither is the model:
- Padding waste. A batch is as slow as its longest sequence. Sorting by length before batching keeps short documents from being padded up to the longest one in the corpus.
- Batch size versus memory.
adaptive_batchescaps onmax_len * batch_sizerather than record count, so a batch of long documents automatically becomes a smaller batch.
padding_efficiency() reports how much of what you processed was real rather than padding. Below
about 0.7, the length distribution is too spread out for the current batch size.
Worth knowing: sorting helps when batch size is the binding constraint. When the token cap binds first, both orderings produce identical batches and sorting buys nothing. Both regimes are asserted in the tests rather than assumed.
Output
Sharded JSONL (gzip) or Parquet. Many medium shards rather than one large file, because shards parallelize across dataloader workers, resume cleanly after a failure, and stream from object storage without a full download.
Both writers emit manifest.json. read_shards() reads it rather than globbing the directory, so
a truncated or partially uploaded dataset raises an error instead of silently yielding fewer records
than you think you have.
Tuning
| Symptom | Change |
|---|---|
| Keeping too much junk | Lower --threshold toward 0.7; raise the word_count minimum |
| Dropping good documents | Raise --threshold; check stats.json for the dominant rule |
| Dedup too slow | Lower --num-perm to 64 |
| Missing obvious duplicates | Raise --bands for higher recall and more candidates to verify |
| Templated pages surviving | --exact-level structural |
| Low padding efficiency | Lower max_batch_tokens, keep sort_by_length=True |
PipelineConfig exposes all of it. See docs/tuning.md and
docs/architecture.md.
Testing
pip install -e ".[dev]"
pytest
112 tests, 92 percent line coverage, no network access and no GPU required. The uncovered remainder
is ParquetShardWriter and SentenceTransformerEncoder, which need pyarrow and torch and are
not installed in CI. They are thin call-throughs; the logic they sit behind is covered.
CI runs on Python 3.10, 3.11, and 3.12, and includes a job that fails if the core ever acquires a runtime dependency.
Documentation
| Guide | For |
|---|---|
| Quickstart | Five minutes, no crawler needed. Real input and real output |
| Architecture | Why the stages are ordered this way, memory ceiling, limitations |
| Tuning | Calibrating thresholds against your own corpus |
| Extending | Swapping the extractor, model, writer, or similarity metric |
| Roadmap | What is next, and what is deliberately not planned |
| Changelog | What changed in each release |
| Contributing | Setup, the rules specific to this codebase |
Contact
Issues and PRs welcome. If an issue is not the right shape, or for commercial use and private corpora, email contact@eprecisio.com.
Maintained by Ehtisham Mubarik (LinkedIn) at Eprecisio Technologies.
Origin
This repository began as a containerized Scrapy deployment (Scrapy, Scrapyd, Postgres, Filebeat,
Jenkins) built for a Swiss real-estate crawl. That crawler is still here under immo_crawl/ as a
working integration example. websieve is the part that was missing: everything between a finished
crawl and a dataset you would actually train on.
License
MIT. See LICENSE.
Project details
Verified details
These details have been verified by PyPIProject links
GitHub Statistics
Maintainers
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file websieve-0.1.1.tar.gz.
File metadata
- Download URL: websieve-0.1.1.tar.gz
- Upload date:
- Size: 42.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
ada91020bdb264227e69e5aab88737c9fd344140369c8f0b41763152914e6ee8
|
|
| MD5 |
700fcc0f4ea119c9e51798291994e051
|
|
| BLAKE2b-256 |
23e2e97dc18c15a3973980b20c797e4d5b304688cb697e345b1966ecce9212ea
|
Provenance
The following attestation bundles were made for websieve-0.1.1.tar.gz:
Publisher:
release.yml on ehtishammubarik/websieve
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
websieve-0.1.1.tar.gz -
Subject digest:
ada91020bdb264227e69e5aab88737c9fd344140369c8f0b41763152914e6ee8 - Sigstore transparency entry: 2278050398
- Sigstore integration time:
-
Permalink:
ehtishammubarik/websieve@39d5775d87c3fbbb165890e14976bb782f4afd87 -
Branch / Tag:
refs/tags/v0.1.1 - Owner: https://github.com/ehtishammubarik
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@39d5775d87c3fbbb165890e14976bb782f4afd87 -
Trigger Event:
push
-
Statement type:
File details
Details for the file websieve-0.1.1-py3-none-any.whl.
File metadata
- Download URL: websieve-0.1.1-py3-none-any.whl
- Upload date:
- Size: 32.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
753d3c74012c47d3e2096c8ab8d75a5327b6f201e45ba61df5e416f9ae4a0937
|
|
| MD5 |
f71c7eda5a0d9ba155d24f905c635192
|
|
| BLAKE2b-256 |
adb5bedcf59006ac118f517f0b593de74c2e66b413cfd9aa97008fe19c6ae682
|
Provenance
The following attestation bundles were made for websieve-0.1.1-py3-none-any.whl:
Publisher:
release.yml on ehtishammubarik/websieve
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
websieve-0.1.1-py3-none-any.whl -
Subject digest:
753d3c74012c47d3e2096c8ab8d75a5327b6f201e45ba61df5e416f9ae4a0937 - Sigstore transparency entry: 2278050434
- Sigstore integration time:
-
Permalink:
ehtishammubarik/websieve@39d5775d87c3fbbb165890e14976bb782f4afd87 -
Branch / Tag:
refs/tags/v0.1.1 - Owner: https://github.com/ehtishammubarik
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@39d5775d87c3fbbb165890e14976bb782f4afd87 -
Trigger Event:
push
-
Statement type:
