Skip to main content

A Python wrapper around TGI and TEI servers

Project description

Py-TXI

PyPI version PyPI - Python Version PyPI - Format Downloads PyPI - License Test

Py-TXI is a Python wrapper around Text-Generation-Inference and Text-Embedding-Inference that enables creating and running TGI/TEI instances through the awesome docker-py in a similar style to Transformers API.

Installation

pip install py-txi

Py-TXI is designed to be used in a similar way to Transformers API. We use docker-py (instead of a dirty subprocess solution) so that the containers you run are linked to the main process and are stopped automatically when your code finishes or fails.

Advantages

  • Easy to use: Py-TXI is designed to be used in a similar way to Transformers API.
  • Automatic cleanup: Py-TXI stops the Docker container when your code finishes or fails.
  • Batched inference: Py-TXI supports sending a batch of inputs to the server for inference.
  • Automatic port allocation: Py-TXI automatically allocates a free port for the Inference server.
  • Configurable: Py-TXI allows you to configure the Inference servers using a simple configuration object.
  • Verbose: Py-TXI streams the logs of the underlying Docker container to the main process so you can debug easily.

Usage

Here's an example of how to use it:

from py_txi import TGI, TGIConfig

llm = TGI(config=TGIConfig(model_id="bigscience/bloom-560m", gpus="0"))
output = llm.generate(["Hi, I'm a language model", "I'm fine, how are you?"])
print("LLM:", output)
llm.close()

Output: LLM: [' student. I have a problem with the following code. I have a class that has a method that', '"\n\n"I\'m fine," said the girl, "but I don\'t want to be alone.']

from py_txi import TEI, TEIConfig

embed = TEI(config=TEIConfig(model_id="BAAI/bge-base-en-v1.5"))
output = embed.encode(["Hi, I'm an embedding model", "I'm fine, how are you?"])
print("Embed:", output)
embed.close()

Output: [array([[ 0.01058742, -0.01588806, -0.03487622, ..., -0.01613717, 0.01772875, -0.02237891]], dtype=float32), array([[ 0.02815401, -0.02892136, -0.0536355 , ..., 0.01225784, -0.00241452, -0.02836569]], dtype=float32)]

That's it! Now you can write your Python scripts using the power of TGI and TEI without having to worry about the underlying Docker containers.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

py-txi-0.10.0.tar.gz (11.3 kB view details)

Uploaded Source

Built Distribution

py_txi-0.10.0-py3-none-any.whl (12.5 kB view details)

Uploaded Python 3

File details

Details for the file py-txi-0.10.0.tar.gz.

File metadata

  • Download URL: py-txi-0.10.0.tar.gz
  • Upload date:
  • Size: 11.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.10.14

File hashes

Hashes for py-txi-0.10.0.tar.gz
Algorithm Hash digest
SHA256 a0c93bb0998e51207e0ea9204e995a83a11b9a5a1f63ebe073253295e7271c1e
MD5 946b918000a81d7cd5d8b7e4e9ad888a
BLAKE2b-256 3809555285b5f0e7c1c56048aba4cffe52c743d03074c3208bb3098f57007cda

See more details on using hashes here.

File details

Details for the file py_txi-0.10.0-py3-none-any.whl.

File metadata

  • Download URL: py_txi-0.10.0-py3-none-any.whl
  • Upload date:
  • Size: 12.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.10.14

File hashes

Hashes for py_txi-0.10.0-py3-none-any.whl
Algorithm Hash digest
SHA256 d667f2fb36427e520df9bc14a719428c75438d59f0cf94a3b9027a6984c788c9
MD5 ec299986e9346368c0e4a4804ac0a9b8
BLAKE2b-256 158090a9bc5956c95db55b09e83156d8e9a4aa8829ca300640369cd00cd98806

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page