crawlerHelper 0.0.1

tools which can easy the process of making crawler

Introduction:

This class can be used in crawler project. And it contain two function:

1. proxies IP

2. hide header

If also used pkl cache to speed up the abstract information from IP Pool cache will be expired in one day

How to use: create new crawlerComponent object, and then,you can use:

1. get_an_ip: get a random IP

2. get_a_header: get a random header

3. updateIpLib: update the ip library (original set as once per day)

methods to use

---

Class Version: 2.0.0 in https://gitlab.com/snippets/1873717