londonaicentre-mesa-local 2.5.3

Serve MESA models locally

MESA local

Serve MESA models locally.

• ⬇️ Downloads weights from S3

• 📦 Unpacks

• 🚀 Serves via a local OpenAI-compatible server

Prerequisites

Software

• Python 3.12

Hardware

• A GPU with >=24GB VRAM (tested on NVIDIA A30)

Configuration

Create a file called .env in the directory where you intend to run this package. Populate it with the details you have been provided with in the following format:

MODEL_NAME=
WEIGHTS_ID=
WEIGHTS_KEY=

(Alternative) S3 URI

Download weights directly from an S3 bucket:

MODEL_NAME=
WEIGHTS_URI=
WEIGHTS_REGION=  # optional, defaults to eu-west-2

(Optional) Caching

Download weights and cache to S3 for faster subsequent downloads:

MODEL_NAME=
WEIGHTS_ID=
WEIGHTS_KEY=
WEIGHTS_URI=
WEIGHTS_REGION=  # optional, defaults to eu-west-2

With this configuration:

• First run: Downloads weights and uploads to S3 cache

• Subsequent runs: Downloads directly from S3 cache (faster)

vLLM configuration

The package provides a set of vLLM configuration files for running a specific model on a specific GPU. In addition to MODEL_NAME, this can be specified by adding GPU to the .env.

Individual vLLM settings can also be overridden by adding them to the .env file:

Setting	Alias	Type	Default
MODEL	MODEL_NAME	str	mesalocal
GPU		str	None
MAX_MODEL_LEN	MODEL_LENGTH	int	41152
ENFORCE_EAGER		bool	False
ENABLE_CHUNKED_PREFILL		bool	True
ENABLE_PREFIX_CACHING		bool	True
GPU_MEMORY_UTILIZATION		float	0.9
MAX_NUM_SEQS		int	256
MAX_NUM_BATCHED_TOKENS		int	None
ENABLE_LOG_REQUESTS		bool	False
UVICORN_LOG_LEVEL		str	warning
HTTP_TIMEOUT_KEEP_ALIVE		int	30

Installation

1. (Recommended) Create a virtual environment and activate it:

   python -m venv .venv
   source .venv/bin/activate
   

2. Install this package: pip install londonaicentre-mesa-local.

Usage

CLI (primary)

1. Note command line arguments:

   Argument	Description
   -v, --verbose	Enable debug output (optional)

2. Start the server as follows: mesalocal [args].

Library (secondary)

1. Import and use the logic of this package as a library:

import asyncio
from mesalocal.weights import Weights
from mesalocal.inferrer import VLLM
vllm_config: VLLMConfig = VLLMConfig() # VLLMConfig(model_name="foo", gpu="bar") to use a vLLM config without a .env file
weights: Weights = Weights(vllm_config.model)
if weights.unpack():
    vllm: VLLM = VLLM(weights.get_model_folder(), vllm_config)
    async def run():
        async for output in vllm.generate(prompt):
            print(output.outputs[0].text)
    asyncio.run(run())

Clients

OpenAI (example with Oncollama)

1. Interact with the server using the OpenAI client in python:

   from openai import OpenAI
   from oncoschema.prompt_builder import PromptBuilder # pip install londonaicentre-oncoschema
   
   client = OpenAI(
       base_url="http://localhost:5000/v1",
       api_key="blank" 
   )
   
   response = client.chat.completions.create(
       model="oncollama3betav01",
       messages=[
           {"role": "system", "content": PromptBuilder().build_main_prompt()},
           {"role": "user", "content": "Diagnosis 01/01/26..."}
       ]
   )
   
   print(response.choices[0].message.content)
   

License

This project uses a proprietary license (see LICENSE).