A simple and efficient web crawler in Python.
Project description
Tiny Web Crawler
A simple and efficient web crawler in Python.
Features
- Crawl web pages and extract links
- Handle relative and absolute URLs
- Save crawl results to a JSON file
- Easy to use and extend
Installation
Install using pip:
pip install tiny-web-crawler
Usage
from tiny_web_crawler.crawler import Spider
root_url = 'http://example.com'
max_links = 2
spider = Spider(root_url, max_links)
spider.start()
Output Format
Crawled output sample for https://github.com
{
"http://github.com": {
"urls": [
"http://github.com/",
"https://githubuniverse.com/",
...
],
"https://github.com/solutions/ci-cd": {
"urls": [
"https://github.com/solutions/ci-cd/",
"https://githubuniverse.com/",
...
]
}
}
}
License
This project is licensed under the GNU GPLv3 License - see the LICENSE file for details.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
tiny_web_crawler-0.1.1.tar.gz
(4.7 kB
view hashes)
Built Distribution
Close
Hashes for tiny_web_crawler-0.1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a6c1030841fa582769fe7dff79514ae71d0af250f768731be42091d0bbe8d100 |
|
MD5 | c9b10da6097836bc46dda28ab92ded32 |
|
BLAKE2b-256 | 4cc7e576e2132815a3ce1e2713c842600dc43ffc06067de7892f173d206f14f2 |