关于这个项目
kkrobots 是一款保证安全爬虫的工具,在爬取任何请求前调用 Parse 对象的 can_crawl() 方法即可判断是否符合 robots.txt 协议。
使用流程
使用流程非常简单,在每次爬虫前调用即可:
from kkrobots import Parse
if __name__ == '__main__':
parse = Parse(
user_agent='your spider',
# 该站点任意链接即可
test_url='https://xxxx.com/xxx/xxx/xxx'
)
can_crawl = parse.can_crawl('https://xxxx.com/xxx/xxx')
# 下方执行你的爬虫逻辑
if can_crawl:
pass
关于作者
微信公众号:Python卡皮巴拉
🌟【Python卡皮巴拉】—— 你的Python修炼秘籍,代码界的“神兽”驾到!🌟
Metadata
Release files for kkrobots 1.0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| kkrobots-1.0.1.tar.gz | 4.6 kB | Without provenance | Details for kkrobots-1.0.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| kkrobots-1.0.1-py3-none-any.whl | Python 3 | none | any | Details for kkrobots-1.0.1-py3-none-any.whl |
Total release size: 11.9 kB
Release files / kkrobots-1.0.1.tar.gz
| Download URL | kkrobots-1.0.1.tar.gz |
|---|---|
| Size | 4.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7afddd96714cec3c18a88749ff927ebc8a5f7eb8bb8b4f29168614374379b3b4
|
|
BLAKE2b-256 checksum How to use checksums |
dbea56d7df88019a6fd22b5643305ee3565c4517874b33a69b3f27fc0d1fc545
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.5
|
Release files / kkrobots-1.0.1-py3-none-any.whl
| Download URL | kkrobots-1.0.1-py3-none-any.whl |
|---|---|
| Size | 7.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0f7c59436f14e3800989796816d7e636eaa1d6c998afcb05298f47014e6657fb
|
|
BLAKE2b-256 checksum How to use checksums |
25c8460d0730f334085c98a239550b6268dcd9f68bd1ffb1e3de55e0cec69e8e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.5
|
Security for kkrobots 1.0.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.