Skip to main content

Distributed script crawler framework.

# Simply connecting redis on the worker side of the library provides an-
# executable power for distributed scripts
# All function executions on the sender end will be piped to redis,
# and the worker end will be pulled out of the pipe to execute.
# Support multi-task simultaneous execution! Each execution maintains a taskid,
# and different tasks maintain their configuration space according to the-
# taskid when they are executed simultaneously.

worker

# if in start_worker.py
import vredis
s = vredis.Worker.from_settings(host='xx.xx.xx.xx',port=6666,password='vredis')
s.start()

# if in bash
C:\Users\Administrator>vredis worker -ho xx.xx.xx.xx -po 6666 -pa vredis -db 0
# if not set param. use defaults param.
# default host localhost
# default port 6379
# default password None
# default db 0

sender

from vredis import pipe

pipe.connect(host='xx.xx.xx.xx',port=6666,password='vredis')
pipe.DEBUG = True # True/False. worker prints on the worker_console.

# very low code intrusion, no decorator or even complete barrier-free execution
# The decorated function becomes a send function and is sent to the task pipeline
@pipe
def some(i):
    import time, random
    rd = random.randint(1,2)
    time.sleep(rd)
    print('use func:{}, rd time:{}'.format(i,rd))
    return 123
    # return a data and wraps them in JSON data and passes them in redis.

@pipe.table('mytable') # if not set table, use "default" as tablename
def some2(i):
    print('use func2:{}'.format(i))
    return 333,444
    # if return is a generator or list or tuple,
    # First, he iterates out the parameters and wraps them in JSON data and passes them in.
    # data collection space use tablename <= default tablename space "default".

for i in range(100):
    some(i) # first send task it will get a taskid. info will log out.
    some2(i)

get_data

from vredis import pipe

pipe.connect(host='xx.xx.xx.xx',port=6666,password='vredis')
for i in pipe.from_table(taskid=26):
    print(i)

# the second param is tablename. default tablename is "default"

Metadata

Release files for vredis.py 1.2.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for vredis.py 1.2.5
File Interpreter ABI Platform Actions
vredis.py-1.2.5-py3-none-any.whl Python 3 none any Details for vredis.py-1.2.5-py3-none-any.whl

Release files / vredis.py-1.2.5-py3-none-any.whl

Download URL vredis.py-1.2.5-py3-none-any.whl
Size 37.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a37d6918e4d40c5df9c763967bff60d8da559f900ba23cb374597e4117e39a54
BLAKE2b-256 checksum
How to use checksums
3f90c14ac8a92d393ed5dd245d7aa85d741e0ac24adff2c9d8b2d6ea857e22dd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via Python-urllib/3.6

Security for vredis.py 1.2.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page