redis-queue-tool 3.3

reids高并发队列(高并发爬虫利器)

包安装:

pip install redis-queue-tool

使用例子:

##### 1.发布消费字符串类型任务
```python
    from redis_queue_tool import RedisQueue
    from redis_queue_tool.RedisQueue import RedisPublish, RedisCustomer


    # redis连接配置
    RedisQueue.redis_host = '127.0.0.1'
    RedisQueue.redis_password = ''
    RedisQueue.redis_port = 6379
    RedisQueue.redis_db = 8

    # 初始化发布队列 queue_name发布队列名称 fliter_rep=True任务自动去重
    redis_pub = RedisPublish(queue_name='test1', fliter_rep=False)

    for zz in range(1, 501):
        redis_pub.publish_redispy_str(str(zz))  # 发布字符串任务


    def print_msg_str(msg):
        print(f"msg_str:{msg}")


    # 消费字符串任务 queue_name消费队列名称
    RedisCustomer(queue_name='test1', consuming_function=print_msg_str, process_num=2, threads_num=100,
                  max_retry_times=5).start_consuming_message()
```

##### 2.发布消费多参数类型任务
```python
    from redis_queue_tool import RedisQueue
    from redis_queue_tool.RedisQueue import RedisPublish, RedisCustomer

    # redis连接配置
    RedisQueue.redis_host = '127.0.0.1'
    RedisQueue.redis_password = ''
    RedisQueue.redis_port = 6379
    RedisQueue.redis_db = 8

    # 初始化发布队列 queue_name发布队列名称 fliter_rep=True任务自动去重
    redis_pub2 = RedisPublish(queue_name='test2', fliter_rep=False)

    for zz in range(1, 501):
        redis_pub2.publish_redispy(c=str(zz), b=str(zz), a=str(zz))  # 写入字典任务 {"c":zz,"b":zz,"a":zz}


    def print_msg_dict(a, b, c):
        print(f"msg_dict:{a},{b},{c}")


    # 消费多参数类型任务 queue_name消费队列名称 is_support_mutil_param=True 消费函数支持多参数
    RedisCustomer(queue_name='test2', consuming_function=print_msg_dict, process_num=2, threads_num=100,
                  max_retry_times=5, is_support_mutil_param=True, qps=50).start_consuming_message()

```

##### 3.批量提交任务

```python
    from redis_queue_tool import RedisQueue
    from redis_queue_tool.RedisQueue import RedisPublish

    # redis连接配置
    RedisQueue.redis_host = '127.0.0.1'
    RedisQueue.redis_password = ''
    RedisQueue.redis_port = 6379
    RedisQueue.redis_db = 8

    result = [str(i) for i in range(1, 501)]
    # 批量提交任务 queue_name提交任务队列名称 max_push_size每次批量提交记录数(默认值50)
    RedisPublish(queue_name='test3', max_push_size=100).publish_redispy_list(result)

```