Scrapy splash wrapper as a standalone library.
Project description
ScrapySplashWrapper
A wrapper that uses scrappy and splash to crawl a website.
Usage
Warning: it requires a splash instance (docker is recommendended).
usage: scraper [-h] [-s SPLASH] -u URL [-d DEPTH] [-o OUTPUT] [-ua USERAGENT]
[--debug]
Crawl a URL.
optional arguments:
-h, --help show this help message and exit
-s SPLASH, --splash SPLASH
Splash URL to use for crawling.
-u URL, --url URL URL to crawl
-d DEPTH, --depth DEPTH
Depth of the crawl.
-o OUTPUT, --output OUTPUT
Output directory
-ua USERAGENT, --useragent USERAGENT
User-Agent to use for crawling
--debug Enable debug mode on scrapy/splash
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file scrapysplashwrapper-1.11.0.tar.gz
.
File metadata
- Download URL: scrapysplashwrapper-1.11.0.tar.gz
- Upload date:
- Size: 9.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.1.13 CPython/3.9.7 Linux/5.13.0-35-generic
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 563475623e0c9f8d09249df20cd65ffff8136b17dbc39ecb6cfb804f7f0a835b |
|
MD5 | 86b11d1e45d939aff4cb57e7128757d4 |
|
BLAKE2b-256 | 18ac183069b4edf74bdd860a07d807da94c3d63843afd837a494ef3f7dbfd9b6 |
File details
Details for the file scrapysplashwrapper-1.11.0-py3-none-any.whl
.
File metadata
- Download URL: scrapysplashwrapper-1.11.0-py3-none-any.whl
- Upload date:
- Size: 11.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.1.13 CPython/3.9.7 Linux/5.13.0-35-generic
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 04653a97ca0f44afd1c65902bf7055f0c476c7356843c76bc3b4098a1acbf693 |
|
MD5 | a57763bb119ea5c19f88c3084f41b455 |
|
BLAKE2b-256 | be470426dfdd492c3676634997aea6e1fd4f7bd07a86ef0139f5bff533bfc5da |