Skip to main content

A lightweight network request framework

Project description

async-request

A lightweight network request framework based on requests & asyncio

install

pip install async_request

usage

Just like scrapy:

from async_request import AsyncSpider, Request


class MySpider(AsyncSpider):
    
    start_urls = ['https://cn.bing.com/']
    
    async def parse(self, response):
        print(response.xpath('//a/@href').get())
        yield Request('https://github.com/financialfly/async-request', callback=self.parse_github)

    def parse_github(self, response):
        yield {'hello': 'github'}
    
    async def process_result(self, result):
        # Process result at here.
        print(result)


if __name__ == '__main__':
    # Run spider
    MySpider().run()

For more detailed control (like: handle cookies, download delay, concurrent requests, max retries, logs settings etc.): (refer to the constructor of the Crawler class):

from async_request import AsyncSpider

class MySpider(AsyncSpider):
    ...

if __name__ == '__main__':
    MySpider(
        handle_cookies=True, 
        download_delay=0,
        concurrent_requests=10,
        max_retries=3,
        log_file='spider.log'
    ).run()

test

Use fetch function to get a response immediately:

from async_request import fetch


def parse():
    response = fetch('https://www.bing.com')
    print(response)
    
   
if __name__ == '__main__':
    parse()

the output will like this:

<Response 200 https://cn.bing.com/>

Use the test decorator is also a method to test spider:

import async_request as ar


@ar.test('https://www.baidu.com')
def parse(response):
    print(response.url, response.status_code)
    
    
if __name__ == '__main__':
    parse()

then run the script, you will see the result:

https://www.baidu.com/ 200

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for async-request, version 0.1888
Filename, size File type Python version Upload date Hashes
Filename, size async-request-0.1888.tar.gz (6.8 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page