Skip to main content

Middleware for proxying scrapy requests through the service proxyfordevelopers.com

Project description

proxydata-scrapy

Расширение для проксирования запросов пауков scrapy через сервис proxyfordevelopers.com

Установка

Установка с помощью pip

pip install proxydata-scrapy

Либо скопируйте middleware класс PFDDownloaderMiddleware в файл middlewares.py своего проекта

Добавьте расширение в переменную DOWNLOADER_MIDDLEWARES в settings.py вашего проекта. Так-же для корретной работы необходимо отключить middlewares RobotsTxtMiddleware и HttpCompressionMiddleware

Добавьте переменную PFD_TOKEN в settings.py вашего проекта. Для получения токена вам необходимо зарегистрироваться в сервисе proxyfordevelopers

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware': None,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': None,
    'proxydata_scrapy.middlewares.PFDDownloaderMiddleware': 800,
}

PFD_TOKEN = 'xyz'

Использование

При необходимости передачи дополнительных параметров в API proxyfordevelopers.com используйте переменную cb_kwargs класса scrapy.http.Request Например:

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
    ]

    def start_requests(self):
        cb_kwargs = {
            'proxy_type': 1,
            'js_enabled': True,
            'country': 'RU',
            'protocol': 1
        }
        for url in self.start_urls:
            yield Request(url, dont_filter=True, cb_kwargs=cb_kwargs)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

proxydata-scrapy-0.0.2b0.tar.gz (3.8 kB view details)

Uploaded Source

Built Distribution

proxydata_scrapy-0.0.2b0-py3-none-any.whl (5.2 kB view details)

Uploaded Python 3

File details

Details for the file proxydata-scrapy-0.0.2b0.tar.gz.

File metadata

  • Download URL: proxydata-scrapy-0.0.2b0.tar.gz
  • Upload date:
  • Size: 3.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/39.0.1 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.3

File hashes

Hashes for proxydata-scrapy-0.0.2b0.tar.gz
Algorithm Hash digest
SHA256 c9d5cf3d92cf05dfc098f99f4df6c0d7b46fd9b38eabe233ca8c8d467fc0ccfe
MD5 bfa15846caf8558cb3a3e95e0d38502e
BLAKE2b-256 1927a9ab3c0caa22b84a03f2268c39aea208f9ddaf5cc3deff4ade07b4104143

See more details on using hashes here.

File details

Details for the file proxydata_scrapy-0.0.2b0-py3-none-any.whl.

File metadata

  • Download URL: proxydata_scrapy-0.0.2b0-py3-none-any.whl
  • Upload date:
  • Size: 5.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/39.0.1 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.3

File hashes

Hashes for proxydata_scrapy-0.0.2b0-py3-none-any.whl
Algorithm Hash digest
SHA256 98b4219f1b9f7a450f59965d3ceaa869ad467a09ba338cf39c8f807d9dffa6ec
MD5 a73b58dad122013de0e7fb1bc72f7928
BLAKE2b-256 07ac41734dc9d254e56ae070e472e0fc494b1d3253b86308b27f04886937a483

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page