WebSleuth - A lightweight, extensible web scraping framework.
Project description
🕵️ WebSleuth
WebSleuth is a lightweight, developer-friendly Python library that simplifies web scraping. It wraps around core tools like requests, BeautifulSoup, user-agent rotation, proxy handling, retry logic, and more—designed to be the developer's alternative to boilerplate scraping setup.
🚀 Features
- ✅ Queue-based URL handling
- ✅ CSS & XPath parser support
- ✅ User-Agent rotation with
fake-useragent - ✅ Proxy support (including free/public proxies)
- ✅ Retry logic with exponential backoff
- ✅ Easy export to JSON or CSV
- ✅ Custom parser middleware support
- ✅ Zero CLI opinionation — integrate into your own code!
📦 Installation
pip install websleuth
git clone https://github.com/yourname/websleuth.git
cd websleuth
pip install .
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file websleuth-0.1.0.tar.gz.
File metadata
- Download URL: websleuth-0.1.0.tar.gz
- Upload date:
- Size: 6.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.1.0 CPython/3.13.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
feca4ff5106fe276d605f29b7b4444a637a609d99a022e319046924d1053af21
|
|
| MD5 |
e2f8a75c19c10babb45d4c356e8f5647
|
|
| BLAKE2b-256 |
375ab5e8a8384e9a88b9f8064dd721e3fb379a60eabff5f1828f583a75020bb1
|
File details
Details for the file websleuth-0.1.0-py3-none-any.whl.
File metadata
- Download URL: websleuth-0.1.0-py3-none-any.whl
- Upload date:
- Size: 7.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.1.0 CPython/3.13.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
39acdb0562df5793e6da3ed6f62abc5bcb7105abd8e3901e2c015080faf760be
|
|
| MD5 |
567f77e3fdb3c934b45748f647b0614e
|
|
| BLAKE2b-256 |
789cc3c01066b5c5572afc9b6496b50c74a4b3b4f36b22dff37df90fd5804544
|
