NightCrawler 0.1.6

Description

The NightCrawler is site crawling/spider tool to gather links at the given domain by walking through the whole site and generating simple sitemap.

Limitations

This tools is just a demo. It’s single-threaded script that walks every page it gets and it’s not optimized for speed.

The script sticks to the url provided and does not dive into subdomains of the given domain even if encounters internal redirect like example.com -> www.example.com

Possible enhancements

• Use multi-threading with thread pools

• Use generators to lower memory footprint and gain a bit more speed

• Make preliminary HEAD request to distinguish between text and binary files

• Check Content-Type and exclude files that are not HTMLs

• Add matchers and sitemap generators for additional sitemap flavour (images, videos, etc.)

• More tests (already included tests are only for the most critical classes)

Installation

pip install NightCrawler

virtualenv .env
. .env/bin/activate
pip install NightCrawler

git clone https://github.com/szczad/NightCrawler.git
cd NightCrawler
virtualenv .env
. .env/bin/activate
pip install -e ./

. .env/bin/activate
python setup.py test

Running the script

nightcrawler --help

nightcrawler <url|domain>

<path_to_virtualenv>/bin/nightcrawler <url|domain>

. .env/bin/activate
nightcrawler <url|domain>