A wrapper engine to enforce request-based, token-based, and concurrency-based limits on kani engines.
Reason this release was yanked:
Broken version
Project description
kani-ratelimits
This is a simple, small package to to enforce request-per-minute (RPM), token-per-minute (TPM), and/or max-concurrency ratelimits before making requests to an underlying engine.
Installation
pip install kani-ratelimits
Usage
from kani.ext.ratelimits import RatelimitedEngine
# limit requests to 10 req/min and 30k tokens/min
inner_engine = ... # your engine here, e.g. `OpenAIEngine(..., model="gpt-4")`
engine = RatelimitedEngine(inner_engine, rpm_limit=10, tpm_limit=30_000)
The RatelimitedEngine
takes the following parameters:
engine
: The engine to wrap.max_concurrency
(int): The maximum number of concurrent requests to serve at once (default unlimited).rpm_limit
(float): The maximum number of requests to serve per rpm_period (default unlimited).rpm_period
(float): The duration, in seconds, of the time period in which to limit the rate. Note that up to rpm_limit requests are allowed within this time period in a burst (default 60s).tpm_limit
(float): The maximum number of tokens to send in requests per tpm_period (default unlimited).tpm_period
(float): The duration, in seconds, of the time period in which to limit the rate. Note that up to tpm_limit tokens are allowed within this time period in a burst (default 60s).
The ratelimiter will ensure that all conditions are met before forwarding the request to the wrapped engine.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
kani_ratelimits-1.0.0.tar.gz
(5.2 kB
view hashes)
Built Distribution
Close
Hashes for kani_ratelimits-1.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3da0a1ce5bef60d1b4d1ad2f2f3e8a3d975c36d06cb56db338f71f8f36f5ad3b |
|
MD5 | 20c02f8cb8307ed760798139c755ccb9 |
|
BLAKE2b-256 | 25b71dad38efcbb971003f026fef8fb0dfa370b7521b486c4fab671749531c0d |