AsyncURLCrawler

AsyncURLCrawler navigates through web pages concurrently by following hyperlinks to collect URLs. AsyncURLCrawler uses BFS algorithm. To make use of it check robots.txt of the domains first.

👉 For complete documentation read here

👉 Source code on Github here

---

Install Pacakge

pip install AsyncURLCrawler

pip install AsyncURLCrawler==<version>

👉 The official page of the project in PyPi.

---

Usage Example in Code

Here is a simple python script to show how to use the package:

import asyncio
import os
from AsyncURLCrawler.parser import Parser
from AsyncURLCrawler.crawler import Crawler
import yaml


async def main():
    parser = Parser(
        delay_start=0.1, 
        max_retries=5, 
        request_timeout=1,
        user_agent="Mozilla",
    )
    crawler = Crawler( 
        seed_urls=["https://pouyae.ir"],
        parser=parser,
        exact=True,
        deep=False,
        delay=0.1,
    )
    result = await crawler.crawl()
    with open(
            os.path.join(output_path, 'result.yaml'), 'w') as file:
        for key in result:
            result[key] = list(result[key])
        yaml.dump(result, file)


if __name__ == "__main__":
    asyncio.run(main())

This is the output for the above code:

https://pouyae.ir:
- https://github.com/PouyaEsmaeili/AsyncURLCrawler
- https://pouyae.ir/images/pouya3.jpg
- https://github.com/PouyaEsmaeili/CryptographicClientSideUserState
- https://github.com/PouyaEsmaeili/RateLimiter
- https://pouyae.ir/
- https://github.com/luizdepra/hugo-coder/
- https://duman.pouyae.ir/
- https://pouyae.ir/projects/
- https://pouyae.ir/images/pouya4.jpg
- https://pouyae.ir/images/pouya5.jpg
- https://pouyae.ir/gallery/
- https://github.com/PouyaEsmaeili
- https://pouyae.ir/blog/
- https://www.linkedin.com/in/pouya-esmaeili-9124b839/
- https://pouyae.ir/about/
- https://stackoverflow.com/users/13118327/pouya-esmaeili?tab=profile
- https://pouyae.ir/contact-me/
- https://github.com/PouyaEsmaeili/SnowflakeID
- https://pouyae.ir/images/pouya2.jpg
- https://github.com/PouyaEsmaeili/gFuzz
- https://linktr.ee/pouyae
- https://gohugo.io/
- https://pouyae.ir/images/pouya1.jpg

👉 There is also a blog post about using AsyncURLCrawler to find malicious URLs in a web page. Read here.

---

Commandline Tool

The script can be customized using the src/cmd/cmd.py file, which accepts various arguments to configure the crawler's behavior:

argument	description
--url	Specifies a list of URLs to crawl. At least one URL must be provided.
--exact	Optional flag; if set, the crawler will restrict crawling to the specified subdomain/domain only. Default is False.
--deep	Optional flag; if enabled, the crawler will explore all visited URLs. Not recommended due to potential resource intensity. If --deep is True, the --exact flag is ignored.
--delay	Sets the delay between consecutive HTTP requests, in seconds.
--output	Specifies the path for the output file, which will be saved in YAML format.

---

Run Commandline Tool in Docker Container 🐳

There is a Dockerfile in src/cmd to run the above-mentioned cmd tool in a docker container.

docker build -t crawler .

docker run -v my_dir:/src/output --name crawler crawler

After execution of the container, the resulting output file will be accessible in the directory named my_dir as defined in the above. To configure the tool based on your needs check the CMD in Dockerfile.

---

Build and Publish to Python Package Index(PyPi)

Requirements:

python3 -m pip install --upgrade build

python3 -m pip install --upgrade twine

👉 For more details check Packaging Python Projects.

Build and upload:

python3 -m build

python3 -m twine upload --repository pypi dist/*

---

Build Documentation with Sphinx

Install packages listed in docs/doc-requirements.txt.

cd docs

pip install -r doc-requirements.txt

make clean

make html

HTML files will be generated in docs/build. Push them the repository and deploy on pages.dev.

---

Workflow

• Branch off, implement features and merge them to main. Remove feature branches.
• Update version in pyproject.toml and push to main.
• Add release tag in Github.
• Build and push the package to PyPi.
• Build documentation and push HTML files to AsyncURLCrawlerDocs repo
• Documentation will be deployed on pages.dev automatically.

---

Contact

Find me @ My Homepage

---

Disclaimer

⚠️ Use at your own risk. The author and contributors are not responsible for any misuse or consequences resulting from the use of this project.

---