reids高并发队列(高并发爬虫利器)
Project description
包安装:
pip install redis-queue-tool
使用实例:
# redis配置连接信息 redis_host = '127.0.0.1' redis_password = '' redis_port = 6379 redis_db = 0 quenen_name = 'test1' redis_pub = RedisPublish(queue_name=quenen_name, max_push_size=5) result = [str(i) for i in range(1, 101)] for zz in result: redis_pub.publish_redispy(a=zz, b=zz, c=zz) # 多线程单条记录写入 redis_pub.publish_redispy_list(result) # 单线程批量写入1 for zz in result: redis_pub.publish_redispy_mutil(zz) # 单线程批量写入2 def print_msg(msg): print(json.loads(msg)) # 多线程消费 redis_customer = RedisCustomer(quenen_name, consuming_function=print_msg, threads_num=100) print(redis_customer.threads_num) redis_customer.start_consuming_message()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
redis-queue-tool-1.4.tar.gz
(1.8 kB
view hashes)
Built Distribution
Close
Hashes for redis_queue_tool-1.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7de8021045b360c2696284ab8ed6fa59b59a49e2f7fdad42cc0ac4deec603d94 |
|
MD5 | 391f85b4f9277f0b8cfc69dc88fd893f |
|
BLAKE2b-256 | 082c482873b7de3abd91d1e798a79d338e7178fcb3fae69620ad324a3b055bc9 |