A Django app to gather the links of a website.
Project description
Crawler is a Django app to help connect to a website and gather as much links as you want.
Detailed documentation is in the “docs” directory.
Quick start
Add “gatherlinks” to your INSTALLED_APPS setting like this:
INSTALLED_APPS = [ ... 'gatherlinks', ]
Include the polls URLconf in your project urls.py like this:
path('crawl/', include('crawl.urls')),
Run python manage.py migrate to create the polls models.
Visit http://127.0.0.1:8000/crawl/ to input the domain name of the website you want to gather it’s links.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
django-web-crawler-0.3.tar.gz
(6.5 kB
view hashes)