Distributed script crawler framework.

# Simply connecting redis on the worker side of the library provides an-
# executable power for distributed scripts
# All function executions on the sender end will be piped to redis,
# and the worker end will be pulled out of the pipe to execute.
# Support multi-task simultaneous execution! Each execution maintains a taskid,
# and different tasks maintain their configuration space according to the-
# taskid when they are executed simultaneously.

worker

# if in start_worker.py
import vredis
s = vredis.Worker.from_settings(host='xx.xx.xx.xx',port=6666,password='vredis')
s.start()

# if in bash
C:\Users\Administrator>vredis worker -ho xx.xx.xx.xx -po 6666 -pa vredis -db 0
# if not set param. use defaults param.
# default host localhost
# default port 6379
# default password None
# default db 0

sender

from vredis import pipe

pipe.connect(host='xx.xx.xx.xx',port=6666,password='vredis')
pipe.DEBUG = True # True/False. worker prints on the worker_console.

# very low code intrusion, no decorator or even complete barrier-free execution
# The decorated function becomes a send function and is sent to the task pipeline
@pipe
def some(i):
    import time, random
    rd = random.randint(1,2)
    time.sleep(rd)
    print('use func:{}, rd time:{}'.format(i,rd))
    return 123
    # return a data and wraps them in JSON data and passes them in redis.

@pipe.table('mytable') # if not set table, use "default" as tablename
def some2(i):
    print('use func2:{}'.format(i))
    return 333,444
    # if return is a generator or list or tuple,
    # First, he iterates out the parameters and wraps them in JSON data and passes them in.
    # data collection space use tablename <= default tablename space "default".

for i in range(100):
    some(i) # first send task it will get a taskid. info will log out.
    some2(i)

get_data

from vredis import pipe

pipe.connect(host='xx.xx.xx.xx',port=6666,password='vredis')
for i in pipe.from_table(taskid=26):
    print(i)

# the second param is tablename. default tablename is "default"

Metadata

Release files for vredis.py 1.2.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for vredis.py 1.2.5
File	Interpreter	ABI	Platform	Reset
vredis.py-1.2.5-py3-none-any.whl 37.2 kB Apr 7, 2019	Python 3	none	any	Details

Release files / vredis.py-1.2.5-py3-none-any.whl

Download URL	vredis.py-1.2.5-py3-none-any.whl
Size	37.2 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`a37d6918e4d40c5df9c763967bff60d8da559f900ba23cb374597e4117e39a54`
BLAKE2b-256 checksum How to use checksums	`3f90c14ac8a92d393ed5dd245d7aa85d741e0ac24adff2c9d8b2d6ea857e22dd`
Upload date	Apr 7, 2019
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`Python-urllib/3.6`