A terribly coded web spider, but useful for recursive API downloads.
Project description
# spidey.py
> Web spiders are usually disliked by websites, but useful for recursive API/page downloads for offline analysis.
## Installation
> Pypi Location: https://pypi.python.org/pypi/spidey.py
- Using Pypi - `pip install spidey`
## Usage
> Run `spidey` for Detailed help.
- `spidey --dir NEW_DIR --filter DOMAIN --url URL [--base BASE_URL]`
- `spidey --dir NEW_DIR --filter DOMAIN --url URL --max MAX_DOWNLOADS`
- Example - `spidey --dir test --filter 'www.google.com' --url 'https://www.google.com/' --max 20`
### More Examples
```
spidey \
-d test \
-f 'www.google.com' \
-u 'https://www.google.com/' \
-b 'https://www.google.com/' \
-hh '{"Accept" : "application/json"}' \
-n 2 \
-m 10 \
-s 5
```
```
spidey \
--dir test \
--filter 'www.google.com' \
--url 'https://www.google.com/'' \ \
--base 'https://www.google.com/
--headers '{"Accept" : "application/json"}' \
--depth 2 \
--max 10 \
--sleep 5
```
> Web spiders are usually disliked by websites, but useful for recursive API/page downloads for offline analysis.
## Installation
> Pypi Location: https://pypi.python.org/pypi/spidey.py
- Using Pypi - `pip install spidey`
## Usage
> Run `spidey` for Detailed help.
- `spidey --dir NEW_DIR --filter DOMAIN --url URL [--base BASE_URL]`
- `spidey --dir NEW_DIR --filter DOMAIN --url URL --max MAX_DOWNLOADS`
- Example - `spidey --dir test --filter 'www.google.com' --url 'https://www.google.com/' --max 20`
### More Examples
```
spidey \
-d test \
-f 'www.google.com' \
-u 'https://www.google.com/' \
-b 'https://www.google.com/' \
-hh '{"Accept" : "application/json"}' \
-n 2 \
-m 10 \
-s 5
```
```
spidey \
--dir test \
--filter 'www.google.com' \
--url 'https://www.google.com/'' \ \
--base 'https://www.google.com/
--headers '{"Accept" : "application/json"}' \
--depth 2 \
--max 10 \
--sleep 5
```
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
spidey.py-0.4.4.tar.gz
(4.0 kB
view hashes)
Built Distribution
Close
Hashes for spidey.py-0.4.4-py2-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f26d52d7f38365260707d6af677b9eeb14b553404177ae2002ed6c5e4235f595 |
|
MD5 | 5c96d7dfaab44d4bf0ca5972b15080fb |
|
BLAKE2b-256 | e012f741dbfc2222e4f53bfc8d8884a128e02af611f484cc359501fde32cbaf2 |