关于这个项目
kkrobots 是一款保证安全爬虫的工具，在爬取任何请求前调用 Parse 对象的 can_crawl() 方法即可判断是否符合 robots.txt 协议。
使用流程
使用流程非常简单，在每次爬虫前调用即可：
from kkrobots import Parse
if __name__ == '__main__':
parse = Parse(
user_agent='your spider',
# 该站点任意链接即可
test_url='https://xxxx.com/xxx/xxx/xxx'
)
can_crawl = parse.can_crawl('https://xxxx.com/xxx/xxx')
# 下方执行你的爬虫逻辑
if can_crawl:
pass
关于作者
微信公众号：Python卡皮巴拉
🌟【Python卡皮巴拉】—— 你的Python修炼秘籍，代码界的“神兽”驾到！🌟
1 maintainer
Metadata
1 maintainer
Release files for kkrobots 1.0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| kkrobots-1.0.1.tar.gz | 4.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| kkrobots-1.0.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 11.9 kB
Release files / kkrobots-1.0.1.tar.gz
| Download URL | kkrobots-1.0.1.tar.gz |
|---|---|
| Size | 4.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7afddd96714cec3c18a88749ff927ebc8a5f7eb8bb8b4f29168614374379b3b4
|
|
BLAKE2b-256 checksum How to use checksums |
dbea56d7df88019a6fd22b5643305ee3565c4517874b33a69b3f27fc0d1fc545
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.5
|
Release files / kkrobots-1.0.1-py3-none-any.whl
| Download URL | kkrobots-1.0.1-py3-none-any.whl |
|---|---|
| Size | 7.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0f7c59436f14e3800989796816d7e636eaa1d6c998afcb05298f47014e6657fb
|
|
BLAKE2b-256 checksum How to use checksums |
25c8460d0730f334085c98a239550b6268dcd9f68bd1ffb1e3de55e0cec69e8e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.5
|
