AsyncURLCrawler navigates through web pages concurrently by following hyperlinks to collect URLs.
Project description
AsyncURLCrawler
AsyncURLCrawler navigates through web pages concurrently by following hyperlinks to collect URLs.
AsyncURLCrawler uses BFS algorithm. To make use of it check robots.txt
of the domains first.
👉 For complete documentation read here
Install Pacakge
pip install AsyncURLCrawler
👉 The official page of the project in PyPi.
Usage Example
Here is a simple python script to show how to use the package:
import asyncio
import os
from AsyncURLCrawler.parser import Parser
from AsyncURLCrawler.crawler import Crawler
import yaml
async def main():
parser = Parser(
delay_start=0.1,
max_retries=5,
request_timeout=1,
user_agent="Mozilla",
)
crawler = Crawler(
seed_urls=["https://pouyae.ir"],
parser=parser,
exact=True,
deep=False,
delay=0.1,
)
result = await crawler.crawl()
with open(
os.path.join(output_path, 'result.yaml'), 'w') as file:
for key in result:
result[key] = list(result[key])
yaml.dump(result, file)
if __name__ == "__main__":
asyncio.run(main())
Build and Publish to Python Package Index(PyPi)
👉 For more details check Packaging Python Projects.
python3 -m build
python3 -m twine upload --repository pypi dist/*
Run with Docker 🐳
docker build -t crawler .
docker run -v my_dir:/src/output --name crawler crawler
After execution of the image, the resulting output file will be accessible in the directory named "my_dir" as defined in the above. To change the output path check the volume in Dockerfile.
To customize the script as your needs, there is cmd.py
file, which accepts various arguments for configuring the crawler:
argument | description |
---|---|
--url |
Specify a list of URLs to be crawled. Ensure at least one URL is provided. |
--exact |
Optional flag; if enabled, the crawler focuses exclusively on the specified subdomain/domain. Default is false. |
--deep |
Optional flag; when activated, the crawler explores all visited URLs (not recommended). If --deep is true, --exact is disregarded. |
--delay |
Define the delay between consecutive HTTP requests (in seconds). |
--output |
Specify the path for the output file, formatted as a YAML file. |
Build Documentation with Sphinx
Install packages listed in doc-requirements.txt
.
cd docs
pip install -r doc-requirements.txt
make clean
make html
HTML files will be generated in docs/build
TODO List
- Add unit tests.
Disclaimer
⚠️ Use at your own risk. The author and contributors are not responsible for any misuse or consequences resulting from the use of this project.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for asyncurlcrawler-0.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aa376e835f515f559fb25bf1c5be7c5b739dfa11d77ac5019f7f7a178c25a2e0 |
|
MD5 | 889d468b97531892df1c6f0325ae1460 |
|
BLAKE2b-256 | 6e0a87ca5e320bde4167a31e6a7f8bf787353b6ea6b0aa53dd217884d15a5b75 |