A simple and efficient web crawler in Python.
Project description
Tiny Web Crawler
A simple and efficient web crawler for Python.
Features
- Crawl web pages and extract links starting from a root URL recursively
- Concurrent workers and custom delay
- Handle relative and absolute URLs
- Designed with simplicity in mind, making it easy to use and extend for various web crawling tasks
Installation
Install using pip:
pip install tiny-web-crawler
Usage
from datacrawl import Spider
from datacrawl import SpiderSettings
settings = SpiderSettings(
root_url = 'http://github.com',
max_links = 2
)
spider = Spider(settings)
spider.start()
# Set workers and delay (default: delay is 0.5 sec and verbose is True)
# If you do not want delay, set delay=0
settings = SpiderSettings(
root_url = 'https://github.com',
max_links = 5,
max_workers = 5,
delay = 1,
verbose = False
)
spider = Spider(settings)
spider.start()
Output Format
Crawled output sample for https://github.com
{
"http://github.com": {
"urls": [
"http://github.com/",
"https://githubuniverse.com/",
"..."
],
"https://github.com/solutions/ci-cd": {
"urls": [
"https://github.com/solutions/ci-cd/",
"https://githubuniverse.com/",
"..."
]
}
}
}
Contributing
Thank you for considering to contribute.
- If you are a first time contributor you can pick a
good-first-issue
and get started. - Please feel free to ask questions.
- Before starting to work on an issue. Please get it assigned to you so that we can avoid multiple people from working on the same issue.
- We are working on doing our first major release. Please check this
issue
and see if anything interests you.
Dev setup
- Install poetry in your system
pipx install poetry
- Clone the repo you forked
- Create a venv or use
poetry shell
- Run
poetry install --with dev
pre-commit install
(see)pre-commit install --hook-type pre-push
Before raising a PR. Please make sure you have these checks covered
- An issue exists or is created which address the PR
- Tests are written for the changes
- All lint/test passes
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
datacrawl-0.6.0.tar.gz
(9.3 kB
view hashes)
Built Distribution
datacrawl-0.6.0-py3-none-any.whl
(10.7 kB
view hashes)
Close
Hashes for datacrawl-0.6.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7be6a754dc5770f99b03ed0d159fd0eee1ff74d83bcc01af0384635a4bfccfbe |
|
MD5 | 072a9bee14905d8128d11c19a9cb9dca |
|
BLAKE2b-256 | 7b395af0e8351c7d657bc8bdaecfc7cc180e0e472aa1f5c2b9b79e2a3fc3d316 |