ScraperLib
_____ _ _ _ / ____| | | (_) | | (___ ___ _ __ __ _ _ __ ___ _ __ | | _| |__ \___ \ / __| '__/ _` | '_ \ / _ \ '__|| | | | '_ \ ____) | (__| | | (_| | |_) | __/ | | |____| | |_) | |_____/ \___|_| \__,_| .__/ \___|_| |______|_|_.__/ | | |_| ============================================================== Starting download of ScraperLib ==============================================================
✨ Features
- Parallel Downloads: Uses Ray to download multiple files simultaneously, maximizing bandwidth and efficiency.
- 403 Avoidance: Rotates user-agents, sets referer headers, and uses session management to avoid being blocked.
- Incremental Mode: Optionally skip files already downloaded.
- Robust State Management: Tracks completed, failed, and skipped downloads with atomic file operations.
- Progress Visualization: Uses tqdm for beautiful progress bars.
- Comprehensive Reporting: Generates JSON reports and visualizations (if matplotlib is installed) of download delays and errors.
- Colorful Console Output: Uses colorama for clear, color-coded logs.
- Dual Logging: Terminal shows only relevant events (e.g.,
[DONE]for successful downloads), while the log file contains all attempts, retries, and errors for full traceability. - Highly Configurable CLI: All parameters (parallelism, chunk size, retry/backoff, output dirs, etc.) can be set via command line.
📦 Installation
Requires Python 3.12+.
-
Clone the repository:
git clone https://github.com/rmonteiro-pereira/Scraper-Lib.git cd Scraper-Lib
-
Install dependencies — the project uses uv and ships a committed
uv.lock, so this reproduces the exact resolved set:uv syncOr with pip, installing the package itself in editable mode:
pip install -e .
Main dependencies:
rayrequeststqdmcoloramabeautifulsoup4matplotlibnumpyportalocker
🚀 Usage
CLI
python -m scraper_lib.cli --url <URL> --patterns .csv .zip --dir data --max-files 10
Main CLI options:
--url: Base URL to scrape for files.--patterns: List of file patterns to match (e.g. .csv .zip).--dir: Download directory.--incremental: Enable incremental download state.--max-files: Limit number of files to download.--max-concurrent: Max parallel downloads.--chunk-size: Chunk size for downloads (e.g. 1gb, 10mb, 8 bytes).--initial-delay: Initial delay between retries (seconds).--max-delay: Maximum delay between retries (seconds).--max-retries: Maximum number of download retries.--state-file: Path for download state file.--log-file: Path for main log file.--report-prefix: Prefix for report files.--headers: Path to JSON file with custom headers.--user-agents: Path to text file with custom user agents (one per line).--disable-logging: Disable all logging for production pipelines.--disable-terminal-logging: Disable terminal logging.--dataset-name: Dataset name for banner.--disable-progress-bar: Disable tqdm progress bar.--output-dir: Directory for report PNGs and JSON.--max-old-logs: Max old log files to keep (default: 25, None disables rotation).--max-old-runs: Max old report/png runs to keep (default: 25, None disables rotation).
See all options with:
python -m scraper_lib --help
Programmatic Usage
from ScraperLib import ScraperLib
scraper = ScraperLib(
base_url: str = "https://example.com/data",
file_patterns: List[str] = [".csv", ".parquet", ".zip"],
download_dir: str = "data",
incremental: bool = True,
max_files: Optional[int] = 2,
max_concurrent: Optional[int] = 16,
chunk_size: Union[str, int] = "10mb",
initial_delay: float = 1.0,
max_delay: float = 60.0,
max_retries: int = 5,
dataset_name: Optional[str] = "MY DATASET"
)
scraper.run()
🛡️ Anti-Blocking Protocols
- User-Agent Rotation: Randomizes user-agent strings on each request and after 403 errors.
- Referer Header: Sets a realistic referer to mimic browser behavior.
- Session Management: Uses a new HTTP session for each attempt.
- Exponential Backoff: Waits longer between retries to avoid rate-limiting.
📊 Reporting
After execution, a summary is printed to the console and a detailed report is saved as a JSON file. If matplotlib is installed, visualizations of download delays are also generated.
🧪 Testing
To run all tests:
pytest tests
📝 Project Structure
.
├── src/
│ ├── __init__.py # Makes src a package
│ ├── scraper_lib.py # Main library
│ ├── DownloadState.py # Download state management
│ ├── CustomLogger.py # Custom logger
├── example.py # Example usage (runnable from root)
├── pyproject.toml # Project metadata and dependencies
├── uv.lock # Locked, reproducible dependency set
├── output/
│ ├── pngs/ # Download delay analysis PNGs
│ └── reports/ # Download reports (JSON)
├── data/ # Downloaded files
├── logs/ # Log files
├── state/ # Download state (auto-generated)
├── tests/ # Unit tests
🤝 Contributing
Pull requests and suggestions are welcome! Please open an issue or submit a PR.
📄 License
This project is licensed under the MIT License.
📬 Contact
Questions or suggestions? Open an issue or contact rmonteiropereira1@gmail.com.
Happy data hunting with ScraperLib! 🚀
1 maintainer
Metadata
1 maintainer
Release files for Scraper-Lib-RMP 0.2.320
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| scraper_lib_rmp-0.2.320.tar.gz | 25.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| scraper_lib_rmp-0.2.320-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 46.7 kB
Release files / scraper_lib_rmp-0.2.320.tar.gz
| Download URL | scraper_lib_rmp-0.2.320.tar.gz |
|---|---|
| Size | 25.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9c23f7f0898340e2fa2ad3f10d0c56387a8b296fc2a5f5d9578a6e2a61754a17
|
|
BLAKE2b-256 checksum How to use checksums |
cd404dabc1f6b4544228ea1adc3a0c053cbe48183589e345bab1e65252577f73
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 31, 2026.
Transparency logRelease files / scraper_lib_rmp-0.2.320-py3-none-any.whl
| Download URL | scraper_lib_rmp-0.2.320-py3-none-any.whl |
|---|---|
| Size | 21.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c1fc7f72207987627ec9820348a2b666428026eb699a18a97833682607124082
|
|
BLAKE2b-256 checksum How to use checksums |
d4e8cbc37c587c6f823a448ea7f3e2106e3883d4cd933924ee4f164b0c2d5a0c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 31, 2026.
Transparency log
