Skip to main content

Scrapy-Domain-Delay

PyPI PyPI - Python Version Build Status

Scrapy-Domain-Delay is a package that lets you set different delay for different website, using the Scrapy framework.

Install

$ pip install scrapy-domain-delay

Usage

Step 1: Extract the domain name from a full url using Python tldextract.

>>> import tldextract
>>> tldextract.extract('https://www.google.com/').domain
'google'

In this example, we would extract "google" as domain name from a full url "https://www.google.com/".

Step 2: Use the following config values in your scrapy settings:

  1. Enable the AutoThrottle extension.

    AUTOTHROTTLE_ENABLED = True
    
  2. Enable the Custom Delay Throttle by adding it to EXTENSIONS.

    EXTENSIONS = {
        'scrapy.extensions.throttle.AutoThrottle': None,
        'scrapy_domain_delay.extensions.CustomDelayThrottle': 300,
    }
    
  3. Add {'domain': 'download delay (in seconds)'} to the DOMAIN_DELAYS.

    something like:

    # set up custom delays per domain
    DOMAIN_DELAYS = {
        'google': 1.0,
        'github': 0.5,
    }
    

Metadata

Release files for scrapy-domain-delay 0.0.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scrapy-domain-delay 0.0.4
File Size Uploaded Provenance Actions
scrapy-domain-delay-0.0.4.tar.gz 2.4 kB Without provenance Details for scrapy-domain-delay-0.0.4.tar.gz

Release files / scrapy-domain-delay-0.0.4.tar.gz

Download URL scrapy-domain-delay-0.0.4.tar.gz
Size 2.4 kB
Tags Source
SHA-256 checksum
How to use checksums
b99d196b967fea5216053d289af74c6cbdd38e9e717cd0958da607f924c10357
BLAKE2b-256 checksum
How to use checksums
d7bdb975b43c6bcd48ae2db3078a9248328760bd994204144ead41797a778aea
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/50.3.0 requests-toolbelt/0.9.1 tqdm/4.49.0 CPython/3.7.1

Security for scrapy-domain-delay 0.0.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page