Skip to main content

Django Web Crawler

Crawler is a Django app to help connect to a website and gather as much links as you want.

Quick start

  1. Add “gatherlinks” to your INSTALLED_APPS setting like this:

    INSTALLED_APPS = [
        ...
        'gatherlinks',
    ]
  2. Import the “main” module like this:

    from gatherlinks.crawler import main
  3. Initialize the StartPoint class like this:

    crawler = main.StartPoint(https://example.com, max_crawl=50, number_of_threads=10)
  4. The StartPoint class can be initialized with three arguments.
    1. homepage (a positional argument of the website to gather it’s link.)

    2. max_crawl (maximum number of links to gather from the website. Default is 50)

    3. number_of_threads (Number of threads to be doing the work simultaneously. Default is 10)

  5. After initialising the class, you can then call the “start” method like this:

    crawler.start()
  6. When the crawler must have finished gathering the link, you can access the gathered links like this:

    crawler.result

That result attribute is a “set” datatype that holds all the links that the crawler could gather. You can then loop through the “crawler.result” and do whatever you want with it (write to file or save to database).

Metadata

Release files for django-web-crawler 0.9

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for django-web-crawler 0.9
File Size Uploaded Provenance Actions
django-web-crawler-0.9.tar.gz 4.5 kB Without provenance Details for django-web-crawler-0.9.tar.gz

Release files / django-web-crawler-0.9.tar.gz

Download URL django-web-crawler-0.9.tar.gz
Size 4.5 kB
Tags Source
SHA-256 checksum
How to use checksums
718c2b15b86f701be27cfe22104f7f299e14351222a7e5cae47ea6dff3dca489
BLAKE2b-256 checksum
How to use checksums
248cd2de600b731cd3e20093eaba925c29810a4c6e851aca9de4d0f476d22b75
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.7.1 importlib_metadata/4.6.4 pkginfo/1.8.2 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.57.0 CPython/3.9.7

Security for django-web-crawler 0.9

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page