Async wrapper for requests / aiohttp, and some python crawler toolkits. Let synchronization code enjoy the performance of asynchronous programming. Read more: https://github.com/ClericPy/torequests.
Project description
Welcome to torequests’s documentation!
Quickstart
To start:
pip install torequests -Urequirements:
requestsfutures # python2aiohttp >= 3.0.5 # python3uvloop # python3optional:
jsonpath_rw_extlxmlobjectpathpsutilfuzzywuzzypython-Levenshteinpyperclip
Examples:
1. Async, threads - make functions asynchronous
from torequests.main import Async, threads import time def use_submit(i): time.sleep(i) result = 'use_submit: %s' % i print(result) return result @threads() def use_decorator(i): time.sleep(i) result = 'use_decorator: %s' % i print(result) return result new_use_submit = Async(use_submit) tasks = [new_use_submit(i) for i in (2, 1, 0) ] + [use_decorator(i) for i in (2, 1, 0)] print([type(i) for i in tasks]) results = [i.x for i in tasks] print(results) # use_submit: 0 # use_decorator: 0 # [<class 'torequests.main.NewFuture'>, <class 'torequests.main.NewFuture'>, <class 'torequests.main.NewFuture'>, <class 'torequests.main.NewFuture'>, <class 'torequests.main.NewFuture'>, <class 'torequests.main.NewFuture'>] # use_submit: 1 # use_decorator: 1 # use_submit: 2 # use_decorator: 2 # ['use_submit: 2', 'use_submit: 1', 'use_submit: 0', 'use_decorator: 2', 'use_decorator: 1', 'use_decorator: 0']
2. tPool - thread pool for async-requests
from torequests.main import tPool from torequests.logs import print_info trequests = tPool() test_url = 'http://p.3.cn' ss = [ trequests.get( test_url, retry=2, callback=lambda x: (len(x.content), print_info(len(x.content)))) for i in range(3) ] # or [i.x for i in ss] trequests.x ss = [i.cx for i in ss] print_info(ss) # [2018-03-18 21:18:09]: 612 # [2018-03-18 21:18:09]: 612 # [2018-03-18 21:18:09]: 612 # [2018-03-18 21:18:09]: [(612, None), (612, None), (612, None)]
3. Requests - aiohttp-wrapper
from torequests.dummy import Requests from torequests.logs import print_info trequests = Requests(frequencies={'p.3.cn': (2, 2)}) ss = [ trequests.get( 'http://p.3.cn', retry=1, timeout=5, callback=lambda x: (len(x.content), print_info(trequests.frequencies))) for i in range(4) ] trequests.x ss = [i.cx for i in ss] print_info(ss) # [2018-03-19 00:57:36]: {'p.3.cn': Frequency(sem=<1/2>, interval=2)} # [2018-03-19 00:57:36]: {'p.3.cn': Frequency(sem=<0/2>, interval=2)} # [2018-03-19 00:57:38]: {'p.3.cn': Frequency(sem=<1/2>, interval=2)} # [2018-03-19 00:57:38]: {'p.3.cn': Frequency(sem=<2/2>, interval=2)} # [2018-03-19 00:57:38]: [(612, None), (612, None), (612, None), (612, None)]
4. utils: some useful crawler toolkits
ClipboardWatcher: watch your clipboard changing.Counts: counter while every time being called.Null: will return self when be called, and alway be False.Regex: Regex Mapper for string -> regex -> object.Saver: simple object persistent toolkit with pickle/json.Timer: timing tool.UA: some common User-Agents for crawler.curlparse: translate curl-string into dict of request.md5: str(obj) -> md5_string.print_mem: show the proc-mem-cost with psutil, use this only for lazinesssss.ptime: %Y-%m-%d %H:%M:%S -> timestamp.ttime: timestamp -> %Y-%m-%d %H:%M:%Sslice_by_size: slice a sequence into chunks, return as a generation of chunks with size.slice_into_pieces: slice a sequence into n pieces, return a generation of n pieces.timeago: show the seconds as human-readable.unique: unique one sequence.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
No source distribution files available for this release.See tutorial on generating distribution archives.
Built Distributions
Close
Hashes for torequests-4.7.13-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 31175b6ea18f7f0759749a6c9e66eb12363c3a4172b011f74421b81cd6e58873 |
|
MD5 | 0bf9f917899239f13d1e95a47cac8809 |
|
BLAKE2b-256 | 1eeb7ebf60737842228ea69bf78491a1f7a0011430abc2d556120a05c8291055 |
Close
Hashes for torequests-4.7.13-py2-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 49ef63e0894c921eb2a31ee1e66caf5b421946f7db5509d5349317b6a8781030 |
|
MD5 | ada1bff352a307c2e3d49ec71c14a275 |
|
BLAKE2b-256 | fd6a75e935960b11519d0b66f7cf8ef0a11e64ff6b073cb4b8919b886c9b3730 |