A simple and efficient web crawler in Python.
Project description
Tiny Web Crawler
A simple and efficient web crawler in Python.
Features
- Crawl web pages and extract links
- Handle relative and absolute URLs
- Save crawl results to a JSON file
- Easy to use and extend
Installation
Install using pip:
pip install tiny-web-crawler
Usage
from tiny_web_crawler.crawler import Spider
root_url = 'http://example.com'
max_links = 2
spider = Spider(root_url, max_links)
spider.start()
Output Format
Crawled output sample for https://github.com
{
"http://github.com": {
"urls": [
"http://github.com/",
"https://githubuniverse.com/",
"..."
],
"https://github.com/solutions/ci-cd": {
"urls": [
"https://github.com/solutions/ci-cd/",
"https://githubuniverse.com/",
"..."
]
}
}
}
License
This project is licensed under the GNU GPLv3 License - see the LICENSE file for details.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
tiny_web_crawler-0.1.4.tar.gz
(5.2 kB
view hashes)
Built Distribution
Close
Hashes for tiny_web_crawler-0.1.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d335a445065b4f5ec8f40aed2aaf211caaf71e152eb348bafe2124f2d4b59b7e |
|
MD5 | 235d5ca1ca9b36cd77d8ff31bce28c20 |
|
BLAKE2b-256 | d961b07e0ccc67ce88dfc9031dc825a3035d8989ae1e56415af6afe277d88d4a |