Tools for VLM-1

vlm-tools provides a convenient set of tools to interact with the VLM-1 API.

Installation

The vlm-tools python package is available on PyPI. You can install it using pip:

pip install vlm-tools

Optionally, if you want to install the torch extra dependencies, you can do so by installing the package with the torch extra:

pip install vlm-tools[torch]

Authentication

The VLM-1 API requires an API key to authenticate requests. You can obtain an API key by signing up on the waitlist. Once you have an API key, you can set it in the environment variable VLM_API_KEY:

export VLM_API_KEY='...'

Usage

Image -> JSON

The python client needs to be configured with your personal API key before you can use it. You can set the API key in the environment variable VLM_API_KEY as described above or pass it to the client constructor.

from vlm_tools.api import vlm

>>> image = Image.open(...)
>>> response_json = vlm(image, domain="document.presentation")
>>> response_json
{
  "description": "...",
  "title": "Differentiated Operating Model",
  "page_number": 7,
  "plots": [
    ...
  ],
  "tables": null,
  "others": [
    ...
  ]
}

Streaming Image -> JSON

You can also stream the image to the API using the stream method. This is useful when you want to stream images to our API sequentially, and one-by-one. We provide some helper functions to help you with sampling unique-images (content-based sampler) from a video stream.

from vlm_tools.video import VideoReader, VideoItertools

>>> itertools = VideoItertools()
>>> video = VideoReader("path/to/video.mp4")
>>> for img in itertools.islice(video, similarity_threshold=0.9)
...     response_json = vlm(img, domain="document.presentation")

Requirements

• Python 3.10+

We currently support Python 3.10+ on Linux and macOS. If you have any questions or issues, please open an issue.