Distributed script crawler framework.
# Simply connecting redis on the worker side of the library provides an-
# executable power for distributed scripts
# All function executions on the sender end will be piped to redis,
# and the worker end will be pulled out of the pipe to execute.
# Support multi-task simultaneous execution! Each execution maintains a taskid,
# and different tasks maintain their configuration space according to the-
# taskid when they are executed simultaneously.
worker
# if in start_worker.py
import vredis
s = vredis.Worker.from_settings(host='xx.xx.xx.xx',port=6666,password='vredis')
s.start()
# if in bash
C:\Users\Administrator>vredis worker -ho xx.xx.xx.xx -po 6666 -pa vredis -db 0
# if not set param. use defaults param.
# default host localhost
# default port 6379
# default password None
# default db 0
sender
from vredis import pipe
pipe.connect(host='xx.xx.xx.xx',port=6666,password='vredis')
pipe.DEBUG = True # True/False. worker prints on the worker_console.
# very low code intrusion, no decorator or even complete barrier-free execution
# The decorated function becomes a send function and is sent to the task pipeline
@pipe
def some(i):
import time, random
rd = random.randint(1,2)
time.sleep(rd)
print('use func:{}, rd time:{}'.format(i,rd))
return 123
# return a data and wraps them in JSON data and passes them in redis.
@pipe.table('mytable') # if not set table, use "default" as tablename
def some2(i):
print('use func2:{}'.format(i))
return 333,444
# if return is a generator or list or tuple,
# First, he iterates out the parameters and wraps them in JSON data and passes them in.
# data collection space use tablename <= default tablename space "default".
for i in range(100):
some(i) # first send task it will get a taskid. info will log out.
some2(i)
get_data
from vredis import pipe
pipe.connect(host='xx.xx.xx.xx',port=6666,password='vredis')
for i in pipe.from_table(taskid=26):
print(i)
# the second param is tablename. default tablename is "default"
Metadata
Release files for vredis.py 1.2.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| vredis.py-1.2.5-py3-none-any.whl | Python 3 | none | any | Details for vredis.py-1.2.5-py3-none-any.whl |
Release files / vredis.py-1.2.5-py3-none-any.whl
| Download URL | vredis.py-1.2.5-py3-none-any.whl |
|---|---|
| Size | 37.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
a37d6918e4d40c5df9c763967bff60d8da559f900ba23cb374597e4117e39a54
|
|
BLAKE2b-256 checksum How to use checksums |
3f90c14ac8a92d393ed5dd245d7aa85d741e0ac24adff2c9d8b2d6ea857e22dd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
Python-urllib/3.6
|
Security for vredis.py 1.2.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.