A simple and efficient web crawler in Python.
Project description
Tiny Web Crawler
A simple and efficient web crawler in Python.
Features
- Crawl web pages and extract links
- Handle relative and absolute URLs
- Save crawl results to a JSON file
- Easy to use and extend
Installation
Install using pip:
pip install tiny-web-crawler
Usage
from tiny_web_crawler.crawler import Spider
root_url = 'http://example.com'
max_links = 2
spider = Spider(root_url, max_links)
spider.start()
Output Format
Crawled output sample for https://github.com
{
"http://github.com": {
"urls": [
"http://github.com/",
"https://githubuniverse.com/",
"..."
],
"https://github.com/solutions/ci-cd": {
"urls": [
"https://github.com/solutions/ci-cd/",
"https://githubuniverse.com/",
"..."
]
}
}
}
License
This project is licensed under the GNU GPLv3 License - see the LICENSE file for details.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
tiny_web_crawler-0.1.2.tar.gz
(5.2 kB
view hashes)
Built Distribution
Close
Hashes for tiny_web_crawler-0.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e5c0a523a23e032b45cb481d4caa2a919013a5ac6405ac0183e5902bf58f7ac1 |
|
MD5 | 760a1802fd2813eded83942577d88ad8 |
|
BLAKE2b-256 | 52cd3fe2c4cf2c6d8995585f997778ca34cc62cbbd630132c394df42700bee87 |