A lightweight network request framework
Project description
async-request
A lightweight network request framework based on requests & asyncio
install
pip install async_request
usage
Just like scrapy:
from async_request import AsyncSpider, Request
class MySpider(AsyncSpider):
start_urls = ['https://cn.bing.com/']
def parse(self, response):
print(response.xpath('//a/@href').get())
yield Request('https://github.com/financialfly/async-request', callback=self.parse_github)
def parse_github(self, response):
yield {'hello': 'github'}
def process_result(self, result):
# Process result at here.
print(result)
if __name__ == '__main__':
# Run spider
MySpider().run()
test
Use fetch
function to get a response immediately:
from async_request import fetch
def parse():
response = fetch('https://www.bing.com')
print(response)
if __name__ == '__main__':
parse()
the output will like this:
<Response 200 https://cn.bing.com/>
Use the test
decorator is also a method to test spider:
import async_request as ar
@ar.test('https://www.baidu.com')
def parse(response):
print(response.url, response.status_code)
if __name__ == '__main__':
parse()
then run the script, you will see the result:
https://www.baidu.com/ 200
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
async-request-0.18.tar.gz
(5.7 kB
view hashes)