async-request 0.16

A lightweight network request framework

async-request

A lightweight network request framework based on requests & asyncio

install

pip install async_request

usage

import async_request as ar

def parse_baidu(response):
    print(response.url, response.status_code)
    yield ar.Request('https://cn.bing.com/', callback=parse_bing)

def parse_bing(response):
    print(response.url, response.status_code)
    print(response.xpath('//a/@href').get())
    yield ar.Request('https://github.com/financialfly/async-request', callback=parse_github)

def parse_github(response):
    print(response.url, response.status_code)
    yield {'hello': 'github'}

def process_result(result):
    print(result)

request_list = [ar.Request(url='https://www.baidu.com', callback=parse_baidu)]
ar.crawl(request_list, result_callback=process_result)

And you'll see the result like this:

https://www.baidu.com/ 200
https://cn.bing.com/ 200
javascript:void(0)
https://github.com/financialfly/async-request 200
{'hello': 'github'}

test

You can test your spider like this:

import async_request as ar

@ar.test('https://www.baidu.com')
def parse_baidu_test(response):
    print(response.url, response.status_code)

then run the script:

https://www.baidu.com/ 200