crawl_requests provides api of User-Agent_pool and proxy_pool.
Project description
1. Feactures:
crawl_requests provides api of User-Agent_pool and proxy_pool.
firstly, common requests; secondly, add user-agent requests; finally, add user-agent and proxy requests.
2. Usage:
>>>from crawl_requests import req,req2
>>>req.req_get(url=’https://www.python.org’,headers={},UA_pool=[],proxy_pool=[])
<Response [200]>
>>>rq2 = req2.Req2() #Need time about ua_pool and proxy_pool.
>>>rq2.all_req(method=’get’,url=’https://www.python.org’)
<Response [200]>
3. Tips:
pip install crawl_requests
req2 updates ua and proxy automatically.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
crawl_requests-2.0.0.tar.gz
(5.9 kB
view hashes)
Built Distribution
crawl_requests-2.0.0-py3.6.egg
(13.6 kB
view hashes)