`async-llms`: A Python Library for Asynchronous LLM Calls

async-llms is a Python library for making asynchronous LLM calls to accelerate LLM evaluation experiments.

Installation

You can install the package using pip:

pip install async-llms

Usage

Set API Key

Set the API key for the LLM provider you want to use.

export OPENAI_API_KEY="your_api_key"
export GOOGLE_API_KEY="your_api_key"
export XAI_API_KEY="your_api_key"

If you are using Google Vertex AI, you can set the project and location using the GOOGLE_PROJECT and GOOGLE_LOCATION environment variables.

export USE_VERTEX_AI="True"  # default is False
export GOOGLE_PROJECT="your_project"
export GOOGLE_LOCATION="your_location"  # "us-central1"

(Optional) Set Timeout

You can set the timeout for the LLM calls using the ASYNC_LLM_TIMEOUT environment variable.

export ASYNC_LLM_TIMEOUT=1800  # 1800 seconds per request

Command Line Interface

You can use the package directly from the command line:

async-llms \
    --api_type "openai" \
    --input_jsonl "path/to/input.jsonl" \
    --output_jsonl "path/to/output.jsonl" \
    --num_parallel_tasks "num_parallel_tasks"

Input Format

The input JSONL file format is identical to the one used in OpenAI's Batch API: https://platform.openai.com/docs/guides/batch

{
    "custom_id": "unique_id_for_this_request",
    "body": {
        // Your LLM request parameters here
    }
}

License

MIT License

Metadata

Release files for async-llms 0.3.7

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for async-llms 0.3.7
File	Size	Uploaded
async_llms-0.3.7.tar.gz	8.3 kB	Oct 1, 2025	Details

Built distribution (wheel)

Table of built distributions (wheels) for async-llms 0.3.7
File	Interpreter	ABI	Platform	Reset
async_llms-0.3.7-py3-none-any.whl 9.9 kB Oct 1, 2025	Python 3	none	any	Details

Total release size: 18.2 kB

Release files / async_llms-0.3.7.tar.gz

Download URL	async_llms-0.3.7.tar.gz
Size	8.3 kB
Tags	Source
SHA-256 checksum How to use checksums	`e3388a04ce301c3185024129fdbfc59518431e82fe4000a61e0ffd91d5386396`
BLAKE2b-256 checksum How to use checksums	`651aa5869abf60c809632667762e9e2e15fe69906bf24043af26f4507eb7d6ad`
Upload date	Oct 1, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.10.16`

Release files / async_llms-0.3.7-py3-none-any.whl

Download URL	async_llms-0.3.7-py3-none-any.whl
Size	9.9 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`7b533c358438b61f2de6babc3abaf2bb280a9bb106e086c620f76168a179374f`
BLAKE2b-256 checksum How to use checksums	`31cb13a37ef765eda1f3df23f4a591641b4e9e5c4ab48bd470bf6ef689ac4743`
Upload date	Oct 1, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.10.16`