crawl_requests provides api of User-Agent_pool and proxy_pool.
Project description
1. Feactures:
crawl_requests provides api of User-Agent_pool and proxy_pool.
firstly, common requests; secondly, add user-agent requests; finally, add user-agent and proxy requests.
2. Usage:
>>>from crawl_requests import req
>>>req.req_get(url=’https://baidu.com’,headers={},UA_pool=[],proxy_pool=[])
<Response [200]>
3. Tips:
pip install crawl_requests
Example: proxy_pool=[{‘http’: ‘62.81.245.12:3128’},{‘https’: ‘210.38.1.147:8080’}]
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
crawl_requests-1.1.1.tar.gz
(2.3 kB
view hashes)