A simple and efficient web crawler in Python.
Project description
Tiny Web Crawler
A simple and efficient web crawler in Python.
Features
- Crawl web pages and extract links
- Handle relative and absolute URLs
- Save crawl results to a JSON file
- Easy to use and extend
Installation
Install using pip:
pip install tiny-web-crawler
Usage
from tiny_web_crawler.crawler import Spider
root_url = 'http://example.com'
max_links = 2
spider = Spider(root_url, max_links)
spider.start()
Output Format
Crawled output sample for https://github.com
{
"http://github.com": {
"urls": [
"http://github.com/",
"https://githubuniverse.com/",
"..."
],
"https://github.com/solutions/ci-cd": {
"urls": [
"https://github.com/solutions/ci-cd/",
"https://githubuniverse.com/",
"..."
]
}
}
}
License
This project is licensed under the GNU GPLv3 License - see the LICENSE file for details.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
tiny_web_crawler-0.1.3.tar.gz
(5.2 kB
view hashes)
Built Distribution
Close
Hashes for tiny_web_crawler-0.1.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | adbb92f7e686452a56d8a07daec54c8156177b53e57143d632660c9412111d63 |
|
MD5 | 861cc8ae68cd010da81ec75cfaf46e61 |
|
BLAKE2b-256 | e223f7c70791f5268d7bf7f5bdcc03fec7ff98317e546f2e8d82bb248ce7f649 |