tools which can easy the process of making crawler
Project description
Introduction:
This class can be used in crawler project. And it contain two function:
1. proxies IP
2. hide header
If also used pkl cache to speed up the abstract information from IP Pool cache will be expired in one day
How to use: create new crawlerComponent object, and then,you can use:
1. get_an_ip: get a random IP
2. get_a_header: get a random header
3. updateIpLib: update the ip library (original set as once per day)
methods to use
Class Version: 2.0.0 in https://gitlab.com/snippets/1873717
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
| Filename, size | File type | Python version | Upload date | Hashes |
|---|---|---|---|---|
| Filename, size crawlerHelper-0.0.1-py3-none-any.whl (4.9 kB) | File type Wheel | Python version py3 | Upload date | Hashes View |
| Filename, size crawlerHelper-0.0.1.tar.gz (3.2 kB) | File type Source | Python version None | Upload date | Hashes View |
Close
Hashes for crawlerHelper-0.0.1-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | ecfe6c986857e366d3d26c40464a83cbd0ad1aaf6dffa3a61455fb9a18dcd40c |
|
| MD5 | 2c2530fe1ef5a3bae9768bcb390fc83a |
|
| BLAKE2-256 | f15460872d7cc17e76f4787dc1025d002344a9e67f60fb19c4c03b1cf5a00528 |
