Web spider and fetcher
Project description
Installation
$ pip install spiderfetch
$ spiderfetch
Usage
Fetching
Fetch all urls matching 2008.*.ogg from a page:
$ spiderfetch http://www.fosdem.org/2008/media/video 2008.*ogg
To dump the urls to a file instead of fetching:
$ spiderfetch http://www.fosdem.org/2008/media/video 2008.*ogg --dump > urls
Spidering
Spider a site to depth 3 while pausing 2 seconds between fetches. The urls that will be considered when spidering must match .*:
$ spiderfetch --host http://en.wikipedia.org --depth 3 --pause 2 '.*'
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
spiderfetch-0.4.9.tar.gz
(18.0 kB
view hashes)
Built Distribution
Close
Hashes for spiderfetch-0.4.9-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a8d26d909299edfe00064ecc4263e86834637b0ce447a0202bdbadd79cd065c5 |
|
MD5 | 61c8a8f4195108b69421e647d6558dd1 |
|
BLAKE2b-256 | 8f910194d07bd954e067019153d07dc9a7a0160331ea793aac6a2168765fe32f |