LBI: LLM Batch Inference
Many LLM providers offer 50% token discounts by using batches. This project wraps each provider's batch API in a unified interface.
Running Tests
To run the unit tests:
uv run pytest
To run the integration tests:
uv run pytest -m integration
Use the verbose flag -v to get more detailed output and use the -n flag to run multiple tests in parallel (-n auto runs one test per CPU core in parallel).
Example:
uv run pytest -n auto -v -m integration
You can also run specific tests and fixtures by keyword:
uv run pytest -n auto -v -m integration -k 'full_batch_lifecycle and mistral'
Publishing to PyPI
-
Bump the version in
[pyproject.toml](./pyproject.toml):version = "0.1.2" -
Build the package:
uv build -
Publish package:
uv publish -
Add tag in Git:
git add pyproject.toml git commit -m "Bump version to 0.1.2" git tag v0.1.2 git push origin main --tags
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file llm_batch_inference-0.1.6.tar.gz.
File metadata
- Download URL: llm_batch_inference-0.1.6.tar.gz
- Upload date:
- Size: 15.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.10.7 {"installer":{"name":"uv","version":"0.10.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
4cbe06ce98a9573f8759e3fa08ac2bcc5cf29b4fb1718bfadbbe18d4e50ebf44
|
|
| MD5 |
95818eea6fb3a9e4a2ae56a8eb258263
|
|
| BLAKE2b-256 |
cfcaddc5ad9572d01397c4f37e9f36ae56d9151dc2b20c7294203986fcdafed9
|
File details
Details for the file llm_batch_inference-0.1.6-py3-none-any.whl.
File metadata
- Download URL: llm_batch_inference-0.1.6-py3-none-any.whl
- Upload date:
- Size: 24.1 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.10.7 {"installer":{"name":"uv","version":"0.10.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9c94592f546dc02bdfc0937450c23011b83dba183482836b5cca0a4963e4fa72
|
|
| MD5 |
f1666d78df3007b4538a4a0be740f77e
|
|
| BLAKE2b-256 |
a4c31b389e5a2013b710c4f374ca39a5238fe2368c22d21a0336220712f87d82
|
