No project description provided
Project description
说明
本包用于使用多线程调接口保存结果
输入一个df格式的原始数据,用每一行生成请求参数,来调用某个接口,分批次保存结果到文件。 结果文件支持fth, csv, xlsx三种格式。
示例
1. 准备数据和函数
# a. 输入数据 df (DataFrame格式)
## df的每一行是一次请求需要的原始数据
# b. 处理输入的函数
import json
def makeReqData(json_str):
# json_str: row.to_json() df的一行数据
# TODO: 使用json_str生成请求参数
json_str = json.loads(json_str)
return json_str
# c. 处理输出的函数
def makeResult(r):
# r: res.json(), 接口返回的json
# TODO: 选取需要保存的字段, 保存为新的dict, 用于写到文件
data = r.get("data")
return data
2. 多线程调用接口保存文件
# 方法一
from multi_request import mreq
m = mreq.MultiRequest()
m.url = "http://127.0.0.1:8080/hello" # 请求接口, 目前只支持 POST 方法
m.makeReqData = makeReqData # 你的生成单次请求数据的函数
m.makeResult = makeResult # (可选) 处理单次返回数据的函数, 生成最终结果字典
m.input_data = df # 原始请求数据, pandas的 DataFrame 格式
m.parallel_batch_size = 20 # (可选) 并发数,默认: 100
m.save_batch_size = 12 # (可选) 每几个保存一个文件,默认: 5000
m.res_format = "fth" # (可选) 默认: fth, 支持格式: fth, csv, xlsx
m.res_dir = "data" # (可选) 保存结果的目录, 默认: ./data
m.run()
# 方法二
from multi_request import mreq
params = {
"url": "http://127.0.0.1:8080/hello",
"makeReqData": makeReqData,
"makeResult": makeResult,
"input_data": df,
"parallel_batch_size": 20,
"save_batch_size": 8,
"res_format": "csv",
"res_dir": "tmp_csv",
}
m = mreq.MultiRequest(**params)
m.run()
# 使用默认参数
from multi_request import mreq
params = {
"url": "http://127.0.0.1:8080/hello",
"makeReqData": makeReqData,
"input_data": df,
}
m = mreq.MultiRequest(**params)
m.run()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
multi_request-0.0.6.tar.gz
(3.3 kB
view details)
Built Distribution
File details
Details for the file multi_request-0.0.6.tar.gz
.
File metadata
- Download URL: multi_request-0.0.6.tar.gz
- Upload date:
- Size: 3.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | c10b30c0b345219a012ef65710f8e79df0bf5094d13601d0559262189ffa66b5 |
|
MD5 | 4e4abfb2501af26924ca286c3dc0251f |
|
BLAKE2b-256 | 9f04c9c4b99336ebb6b7a4cbd2d0670a307a1df0c37e27b37ee01573d28bedf1 |
File details
Details for the file multi_request-0.0.6-py3-none-any.whl
.
File metadata
- Download URL: multi_request-0.0.6-py3-none-any.whl
- Upload date:
- Size: 3.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 90d1dcc85f0e938e2ef488798580858cbc5c6f6ef0d366dea59912a937d58295 |
|
MD5 | ad3acb53ec03ba5a91dfd9274c6738e5 |
|
BLAKE2b-256 | 29c975444453d87d3f0f7c27f274ab7b30473ea659fae868244353e2bb5df119 |