foot
Web Crawling Project
Foot is a library that fetches a list of URLs and silly walks through each site to gather information.
Usage
Install
pip install foot
foot (cli)
-uURL(s) (encapsulated in quotes, separated by commas)-ffilename (list of urls on new lines)
Options (not required)
-c: Specify chunk size (default=10)--recursive: Enable one level of recursion
Examples:
Array of URLS:
foot -u 'http://example.com, http://example1.com'
Array of URLS with Options:
foot 'http://example.com, http://example1.com' -c 5 --recursive
File:
foot -f 'example.txt
File with Options:
foot -f 'example.txt' -c 5 --recursive
Module
Import:
import foot
Functions
The get function takes an array of URLS and options.
foot.get(["http://www.example`.com", "http://www.example2.com"], options);
=> Data output in ./foot/url.json and foot-date.txt
The file function takes a filename and options.
foot.file("./test.txt", recursive=True)
=> Data output in ./foot/url.json and foot-date.txt
1 maintainer
Metadata
1 maintainer
Release files for foot 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| foot-0.1.0.tar.gz | 10.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| foot-0.1.0-py2.py3-none-any.whl | Python 2, Python 3 | none | any | Details |
Total release size: 14.9 kB
Release files / foot-0.1.0.tar.gz
| Download URL | foot-0.1.0.tar.gz |
|---|---|
| Size | 10.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
14a854c995c4895545940646d23e8320824fff0a60f54cd9b48e17b353f5ef20
|
|
BLAKE2b-256 checksum How to use checksums |
bbd880a24bc1542f308d77e448578b38954d11e03ec3af8f994fbc924b28503f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.12.1 pkginfo/1.4.2 requests/2.20.1 setuptools/40.5.0 requests-toolbelt/0.8.0 tqdm/4.28.1 CPython/3.7.1
|
Release files / foot-0.1.0-py2.py3-none-any.whl
| Download URL | foot-0.1.0-py2.py3-none-any.whl |
|---|---|
| Size | 4.6 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
36fc882089f97aab770d87472ae6814ffb30b3092625db20f595fb94db6ad1bf
|
|
BLAKE2b-256 checksum How to use checksums |
debcf7b849b6113f4fd2b8847c377bd1d6db494be33d61317205f8e7732a39b3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.12.1 pkginfo/1.4.2 requests/2.20.1 setuptools/40.5.0 requests-toolbelt/0.8.0 tqdm/4.28.1 CPython/3.7.1
|
