A lightweight network request framework
Project description
async-request
A lightweight network request framework based on requests & asyncio
install
pip install async_request
usage
import async_request as ar
def parse_baidu(response):
print(response.url, response.status_code)
yield ar.Request('https://cn.bing.com/', callback=parse_bing)
def parse_bing(response):
print(response.url, response.status_code)
print(response.xpath('//a/@href').get())
yield ar.Request('https://github.com/financialfly/async-request', callback=parse_github)
def parse_github(response):
print(response.url, response.status_code)
yield {'hello': 'github'}
def process_result(result):
print(result)
request_list = [ar.Request(url='https://www.baidu.com', callback=parse_baidu)]
ar.crawl(request_list, result_callback=process_result)
And you'll see the result like this:
https://www.baidu.com/ 200
https://cn.bing.com/ 200
javascript:void(0)
https://github.com/financialfly/async-request 200
{'hello': 'github'}
test
You can test your spider like this:
import async_request as ar
@ar.test('https://www.baidu.com')
def parse_baidu_test(response):
print(response.url, response.status_code)
then run the script:
https://www.baidu.com/ 200
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
async-request-0.16.tar.gz
(4.5 kB
view hashes)