Auto Generated by os-scrapy-cookiecutter
Project description
os-scrapy-aio-resolver
This project provide a async DNS resolver.
Install
pip install os-scrapy-aio-resolver
You can run example spider directly in the project root path
scrapy crawl example
Usage
Settings
-
enable DNS resolver in the project settings file:
DNS_RESOLVER = "os_scrapy_aio_resolver.AsyncResolver"
-
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
or you can use os-scrapy command line to start crawling with
-r asyncio
os-scrapy crawl example -r asyncio
-
enable/disable DNS cache, set cache size
DNSCACHE_ENABLED = True DNSCACHE_SIZE = 10000
Unit Tests
sh scripts/test.sh
License
MIT licensed.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Close
Hashes for os_scrapy_aio_resolver-0.0.3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5bffaf60f46221a1e3b338bf4264259cb23497e92489c2d3a8f84cfbe75710ce |
|
MD5 | 89d3dd38beb38c10121baa07fc602fd9 |
|
BLAKE2b-256 | ed7957b5da3ed4ea6089f000810a28ac9a148103ee01048f38b6dd2988f14fcb |