CrawLsy-Spider
简介
CrawLsy-Spider 是一个基于 Redis 的分布式任务系统，旨在极度简化任务的提交和管理。
安装
- 确保已安装 Python 3.9 或更高版本。
- 安装依赖库：
pip install crawlsy-spider
使用方法
在 task.py 中编写任务逻辑
import requests
def task_func(url):
return requests.get(url).text
在 produce.py 提交任务
from crawlsy_spider import CrawLsy
from task import task_func # 导入test函数
with CrawLsy(
name="test",
host="127.0.0.1",
db=0,
password="test",
) as craw:
job_id = craw.put(task_func, 'https://baidu.com')
工作节点部署
python consumers.py
运行生产节点
由于框架是生产消费分离模式，所以在多服务器（集群中启动 worker），此时服务并不能运行，还需要在新启动一个节点用来启动生产服务
python producer.py
开源协议
GNU General Public License v2.0
Crawlsy 是自由软件，您可以根据自由软件基金会发布的 GNU 通用公共许可证（版本 2）的条款重新分发或修改它。
发布此程序是希望它有用，但不提供任何保证；甚至没有对适销性或特定用途适用性的暗示保证。有关更多详细信息，请参阅 GNU 通用公共许可证。
您应该已经收到了 GNU 通用公共许可证的副本；如果没有，请参阅 http://www.gnu.org/licenses/。
1 maintainer
Metadata
1 maintainer
Release files for crawlsy-spider 0.1.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| crawlsy_spider-0.1.3.tar.gz | 11.0 kB | Without provenance | Details for crawlsy_spider-0.1.3.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| crawlsy_spider-0.1.3-py3-none-any.whl | Python 3 | none | any | Details for crawlsy_spider-0.1.3-py3-none-any.whl |
Total release size: 23.4 kB
Release files / crawlsy_spider-0.1.3.tar.gz
| Download URL | crawlsy_spider-0.1.3.tar.gz |
|---|---|
| Size | 11.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c0043d751c5414cc2c73482ccf5b483f971f07b2ce8a5fe47072d3c607e88a3f
|
|
BLAKE2b-256 checksum How to use checksums |
98737e791cb1d56300f4ac3b0eae3c317e353a75000dba1c0261f93b7e53894b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.0.1 CPython/3.9.21 Linux/6.8.0-1020-azure
|
Release files / crawlsy_spider-0.1.3-py3-none-any.whl
| Download URL | crawlsy_spider-0.1.3-py3-none-any.whl |
|---|---|
| Size | 12.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
256eeb8c2122a5e6e3daf3442c9c4431754a11d863322dec05ab14214e6525da
|
|
BLAKE2b-256 checksum How to use checksums |
86122485b9f20ae4b11da77ec60b3fc628b5bb3bee88df1c6ae6854de41e70d9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.0.1 CPython/3.9.21 Linux/6.8.0-1020-azure
|
Security for crawlsy-spider 0.1.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
