reids高并发队列(高并发爬虫利器)
Project description
包安装:
pip install redis-queue-tool
使用例子:
##### 1.发布消费字符串类型任务 from redis_queue_tool import RedisQueue from redis_queue_tool.RedisQueue import RedisPublish, RedisCustomer # redis连接配置 RedisQueue.init_redis_config(host='127.0.0.1', password='', port=6379, db=8) for zz in range(1, 501): # 发布字符串任务 queue_name发布队列名称 fliter_rep=True任务自动去重(默认False) RedisPublish(queue_name='test1', fliter_rep=False).publish_redispy_str(str(zz)) def print_msg_str(msg): print(f"msg_str:{msg}") # 消费字符串任务 queue_name消费队列名称 max_retry_times错误最大重试次数 RedisCustomer(queue_name='test1', consuming_function=print_msg_str, process_num=2, threads_num=100, max_retry_times=5).start_consuming_message() ##### 2.发布消费多参数类型任务 from redis_queue_tool import RedisQueue from redis_queue_tool.RedisQueue import RedisPublish, RedisCustomer # redis连接配置 RedisQueue.init_redis_config(host='127.0.0.1', password='', port=6379, db=8) for zz in range(1, 501): # 发布多参数任务 RedisPublish(queue_name='test2').publish_redispy(c=str(zz), b=str(zz), a=str(zz)) def print_msg_dict(a, b, c): print(f"msg_dict:{a},{b},{c}") # 消费多参数类型任务 queue_name消费队列名称 is_support_mutil_param=True消费函数支持多参数(默认False) qps每秒消费任务数 RedisCustomer(queue_name='test2', consuming_function=print_msg_dict, process_num=2, threads_num=100, max_retry_times=5, is_support_mutil_param=True, qps=50).start_consuming_message() ##### 3.批量提交任务 from redis_queue_tool import RedisQueue from redis_queue_tool.RedisQueue import RedisPublish # redis连接配置 RedisQueue.init_redis_config(host='127.0.0.1', password='', port=6379, db=8) result = [str(i) for i in range(1, 501)] # 批量提交任务 queue_name提交任务队列名称 max_push_size每次批量提交记录数(默认值50) RedisPublish(queue_name='test3', max_push_size=100).publish_redispy_list(result)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
redis-queue-tool-3.7.tar.gz
(6.5 kB
view hashes)
Built Distribution
Close
Hashes for redis_queue_tool-3.7-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 384a3065724d2c962648a1ec7fcacba9c50ce54110f800a66378654c889f6503 |
|
MD5 | 7e1c41a8a29d2a8ad5b03228c30e9e3e |
|
BLAKE2b-256 | 6ea3911e606fd153b5a549dcb7fc5bfc1f8bf2b2abc4131725c98c853c21732c |