async-request
A lightweight network request framework based on requests & asyncio
install
pip install async_request
usage
Just like scrapy:
from async_request import AsyncSpider, Request
class MySpider(AsyncSpider):
start_urls = ['https://cn.bing.com/']
async def parse(self, response):
print(response.xpath('//a/@href').get())
yield Request('https://github.com/financialfly/async-request', callback=self.parse_github)
def parse_github(self, response):
yield {'hello': 'github'}
async def process_result(self, result):
# Process result at here.
print(result)
if __name__ == '__main__':
# Run spider
MySpider().run()
For more detailed control (like: handle cookies, download delay, concurrent requests, max retries, logs settings etc.): (refer to the constructor of the Crawler class):
from async_request import AsyncSpider
class MySpider(AsyncSpider):
...
if __name__ == '__main__':
MySpider(
handle_cookies=True,
download_delay=0,
concurrent_requests=10,
max_retries=3,
log_file='spider.log'
).run()
test
Use fetch function to get a response immediately:
from async_request import fetch
def parse():
response = fetch('https://www.bing.com')
print(response)
if __name__ == '__main__':
parse()
the output will like this:
<Response 200 https://cn.bing.com/>
Use the test decorator is also a method to test spider:
import async_request as ar
@ar.test('https://www.baidu.com')
def parse(response):
print(response.url, response.status_code)
if __name__ == '__main__':
parse()
then run the script, you will see the result:
https://www.baidu.com/ 200
1 maintainer
Release files for async-request 0.1888
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| async-request-0.1888.tar.gz | 6.8 kB | Details |
Release files / async-request-0.1888.tar.gz
| Download URL | async-request-0.1888.tar.gz |
|---|---|
| Size | 6.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
23306267536c911c15f1c5f2d5b08fc2378bbcd723b915cb4c3dffb4e2cc1df8
|
|
BLAKE2b-256 checksum How to use checksums |
22f20d0bf3be4ce2bdd6b91038756f0aa12808125a22dc2e37d0f270dd19c190
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.13.0 pkginfo/1.5.0.1 requests/2.18.4 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.33.0 CPython/3.7.2
|
