Skip to main content

No project description provided

Project description

说明

本包用于使用多线程调接口保存结果

输入一个df格式的原始数据,用每一行生成请求参数,来调用某个接口,分批次保存结果到文件。 结果文件支持fth, csv, xlsx三种格式。

示例

1. 准备数据和函数

# a. 输入数据 df (DataFrame格式)
## df的每一行是一次请求需要的原始数据

# b. 处理输入的函数
import json
def makeReqData(json_str):
    # json_str: row.to_json() df的一行数据
    # TODO: 使用json_str生成请求参数
    json_str = json.loads(json_str)
    return json_str

# c. 处理输出的函数
def makeResult(r):
    # r: res.json(), 接口返回的json
    # TODO: 选取需要保存的字段, 保存为新的dict, 用于写到文件
    data = r.get("data")
    return data

2. 多线程调用接口保存文件

# 方法一
from multi_request import mreq

m = mreq.MultiRequest()
m.url = "http://127.0.0.1:8080/hello"  # 请求接口, 目前只支持 POST 方法
m.makeReqData = makeReqData  # 你的生成单次请求数据的函数
m.makeResult = makeResult  # (可选) 处理单次返回数据的函数, 生成最终结果字典
m.input_data = df  # 原始请求数据, pandas的 DataFrame 格式
m.parallel_batch_size = 20  # (可选) 并发数,默认: 100
m.save_batch_size = 12  # (可选) 每几个保存一个文件,默认: 5000
m.res_format = "fth"  # (可选) 默认: fth, 支持格式: fth, csv, xlsx
m.res_dir = "data"  # (可选) 保存结果的目录, 默认: ./data
m.run()
# 方法二
from multi_request import mreq

params = {
    "url": "http://127.0.0.1:8080/hello",
    "makeReqData": makeReqData,
    "makeResult": makeResult,
    "input_data": df,
    "parallel_batch_size": 20,
    "save_batch_size": 8,
    "res_format": "csv",
    "res_dir": "tmp_csv",
}
m = mreq.MultiRequest(**params)
m.run()
# 使用默认参数
from multi_request import mreq

params = {
    "url": "http://127.0.0.1:8080/hello",
    "makeReqData": makeReqData,
    "input_data": df,
}
m = mreq.MultiRequest(**params)
m.run()

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

multi_request-0.0.6.tar.gz (3.3 kB view details)

Uploaded Source

Built Distribution

multi_request-0.0.6-py3-none-any.whl (3.4 kB view details)

Uploaded Python 3

File details

Details for the file multi_request-0.0.6.tar.gz.

File metadata

  • Download URL: multi_request-0.0.6.tar.gz
  • Upload date:
  • Size: 3.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.8.12

File hashes

Hashes for multi_request-0.0.6.tar.gz
Algorithm Hash digest
SHA256 c10b30c0b345219a012ef65710f8e79df0bf5094d13601d0559262189ffa66b5
MD5 4e4abfb2501af26924ca286c3dc0251f
BLAKE2b-256 9f04c9c4b99336ebb6b7a4cbd2d0670a307a1df0c37e27b37ee01573d28bedf1

See more details on using hashes here.

File details

Details for the file multi_request-0.0.6-py3-none-any.whl.

File metadata

File hashes

Hashes for multi_request-0.0.6-py3-none-any.whl
Algorithm Hash digest
SHA256 90d1dcc85f0e938e2ef488798580858cbc5c6f6ef0d366dea59912a937d58295
MD5 ad3acb53ec03ba5a91dfd9274c6738e5
BLAKE2b-256 29c975444453d87d3f0f7c27f274ab7b30473ea659fae868244353e2bb5df119

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page