sitecrawl
Simple Python module to crawl a website and extract URLs.
Installation
Using pip:
pip3 install sitecrawl
sitecrawl --helpOr build from sources:
# Clone project
git clone https://github.com/gabfl/sitecrawl && cd sitecrawl
# Installation
pip3 install .Usage
CLI
sitecrawl --url https://www.yahoo.com/ --depth 2 --max 4 --verbose->
* Found 4 internal URLs https://www.yahoo.com https://www.yahoo.com/entertainment https://www.yahoo.com/lifestyle https://www.yahoo.com/plus * Found 5 external URLs https://mail.yahoo.com/ https://news.yahoo.com/ https://finance.yahoo.com/ https://sports.yahoo.com/ https://shopping.yahoo.com/ * Skipped 0 URLs
As a module
Basic example:
from sitecrawl import crawl
crawl.base_url = 'https://www.yahoo.com'
crawl.deep_crawl(depth=2)
print('Internal URLs:', crawl.get_internal_urls())
print('External URLs:', crawl.get_external_urls())
print('Skipped URLs:', crawl.get_skipped_urls())A more detailed example is available in example.py.
1 maintainer
Metadata
1 maintainer
Release files for sitecrawl 1.0.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| sitecrawl-1.0.5.tar.gz | 5.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| sitecrawl-1.0.5-py2.py3-none-any.whl | Python 2, Python 3 | none | any | Details |
Total release size: 11.4 kB
Release files / sitecrawl-1.0.5.tar.gz
| Download URL | sitecrawl-1.0.5.tar.gz |
|---|---|
| Size | 5.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
203417c73038f3beb7ad185985ffe7fe0aef2c6d0b88a6bfbcab886e4350eb07
|
|
BLAKE2b-256 checksum How to use checksums |
99d6cf003181dc0a933c51e8274ca964adf7d6a63b2cbe1c2c42819e4aaf0d5e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.3
|
Release files / sitecrawl-1.0.5-py2.py3-none-any.whl
| Download URL | sitecrawl-1.0.5-py2.py3-none-any.whl |
|---|---|
| Size | 6.0 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
39d8e219c7e75252395ef91666626c8c4fb72657f632481af3f4e6db019f1e60
|
|
BLAKE2b-256 checksum How to use checksums |
aaf556d45dffb05dd630a0dc062859088e99f0178e3c39bcd3332814d9ff6523
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.3
|
