Skip to main content

A lightweight network request framework

Project description

async-request

A lightweight network request framework based on requests & asyncio

install

pip install async_request

usage

Just like scrapy:

from async_request import AsyncSpider, Request


class MySpider(AsyncSpider):
    
    start_urls = ['https://cn.bing.com/']
    
    async def parse(self, response):
        print(response.xpath('//a/@href').get())
        yield Request('https://github.com/financialfly/async-request', callback=self.parse_github)

    def parse_github(self, response):
        yield {'hello': 'github'}
    
    async def process_result(self, result):
        # Process result at here.
        print(result)


if __name__ == '__main__':
    # Run spider
    MySpider().run()

For more detailed control (like: handle cookies, download delay, concurrent requests, max retries, logs settings etc.): (refer to the constructor of the Crawler class):

from async_request import AsyncSpider

class MySpider(AsyncSpider):
    ...

if __name__ == '__main__':
    MySpider(
        handle_cookies=True, 
        download_delay=0,
        concurrent_requests=10,
        max_retries=3,
        log_file='spider.log'
    ).run()

test

Use fetch function to get a response immediately:

from async_request import fetch


def parse():
    response = fetch('https://www.bing.com')
    print(response)
    
   
if __name__ == '__main__':
    parse()

the output will like this:

<Response 200 https://cn.bing.com/>

Use the test decorator is also a method to test spider:

import async_request as ar


@ar.test('https://www.baidu.com')
def parse(response):
    print(response.url, response.status_code)
    
    
if __name__ == '__main__':
    parse()

then run the script, you will see the result:

https://www.baidu.com/ 200

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

async-request-0.1888.tar.gz (6.8 kB view details)

Uploaded Source

File details

Details for the file async-request-0.1888.tar.gz.

File metadata

  • Download URL: async-request-0.1888.tar.gz
  • Upload date:
  • Size: 6.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.18.4 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.33.0 CPython/3.7.2

File hashes

Hashes for async-request-0.1888.tar.gz
Algorithm Hash digest
SHA256 23306267536c911c15f1c5f2d5b08fc2378bbcd723b915cb4c3dffb4e2cc1df8
MD5 f692093fae34f06f370b84818bb59bf4
BLAKE2b-256 22f20d0bf3be4ce2bdd6b91038756f0aa12808125a22dc2e37d0f270dd19c190

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page