londonaicentre-mesa-local 2.10.0

Serve MESA models locally

MESA local

Serve MESA models locally.

• ⬇️ Downloads weights from S3

• 📦 Unpacks

• 🚀 Serves via a local OpenAI-compatible server

Prerequisites

Software

• Python 3.12

Hardware

• A GPU with >=24GB VRAM (tested on NVIDIA A30).
• An NVIDIA driver supporting CUDA >=13; hosts limited to CUDA 12.x need the alternative install in CUDA 12.x hosts below.

Configuration

Create a file called .env in the directory where you intend to run this package. Populate it with the details you have been provided with in the following format:

MODEL_NAME=
WEIGHTS_ID=
WEIGHTS_KEY=

(Alternative) S3 URI

Download weights directly from an S3 bucket:

MODEL_NAME=
WEIGHTS_URI=
WEIGHTS_REGION=  # optional, defaults to eu-west-2

(Optional) Caching

Download weights and cache to S3 for faster subsequent downloads:

MODEL_NAME=
WEIGHTS_ID=
WEIGHTS_KEY=
WEIGHTS_URI=
WEIGHTS_REGION=  # optional, defaults to eu-west-2

With this configuration:

• First run: Downloads weights and uploads to S3 cache

• Subsequent runs: Downloads directly from S3 cache (faster)

vLLM configuration

The package provides a set of vLLM configuration files for running a specific model on a specific GPU. In addition to MODEL_NAME, this can be specified by adding GPU to the .env.

Individual vLLM settings can also be overridden by adding them to the .env file:

Setting	Alias	Type	Default
MODEL	MODEL_NAME	str	mesalocal
GPU		str	None
MAX_MODEL_LEN	MODEL_LENGTH	int	41152
ENFORCE_EAGER		bool	False
ENABLE_CHUNKED_PREFILL		bool	True
ENABLE_PREFIX_CACHING		bool	True
GPU_MEMORY_UTILIZATION		float	0.9
MAX_NUM_SEQS		int	256
MAX_NUM_BATCHED_TOKENS		int	None
ENABLE_LOG_REQUESTS		bool	False
UVICORN_LOG_LEVEL		str	warning
HTTP_TIMEOUT_KEEP_ALIVE		int	30

Installation

1. (Recommended) Create a virtual environment and activate it:

   python -m venv .venv
   source .venv/bin/activate
   

2. Install this package: pip install londonaicentre-mesa-local.

CUDA 12.x hosts

1. Install uv.

2. Create a uv-managed environment and install the CUDA 12.9 build (last supported by the current vLLM version) in place of the default above:

   uv venv --python 3.12 .venv && source .venv/bin/activate
   uv pip install --torch-backend=cu129 \
       "vllm @ https://github.com/vllm-project/vllm/releases/download/v0.24.0/vllm-0.24.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl" \
       londonaicentre-mesa-local
   

3. (Optional) Forward compatibility is required if the current CUDA version is below 12.9; a driver already at 12.9 runs these wheels directly and can skip this step. Where needed, unpack cuda-compat-12-9 (no root needed; download the .deb from NVIDIA's CUDA repo, copying it onto the host first if egress is restricted), prepend it to the library path in the shell that will run mesalocal, and verify (expect True 12.9):

   dpkg-deb -x cuda-compat-12-9_575.57.08-0ubuntu1_amd64.deb ~/cuda-compat
   export LD_LIBRARY_PATH=$HOME/cuda-compat/usr/local/cuda-12.9/compat:$LD_LIBRARY_PATH
   python -c "import torch; print(torch.cuda.is_available(), torch.version.cuda)"
   

Usage

CLI (primary)

1. Note command line arguments:

   Argument	Description
   -v, --verbose	Enable debug output (optional)

2. Start the server as follows: mesalocal [args].

Library (secondary)

1. Import and use the logic of this package as a library:

import asyncio
from mesalocal.weights import Weights
from mesalocal.inferrer import VLLM
vllm_config: VLLMConfig = VLLMConfig() # VLLMConfig(model_name="foo", gpu="bar") to use a vLLM config without a .env file
weights: Weights = Weights(vllm_config.model)
if weights.unpack():
    vllm: VLLM = VLLM(weights.get_model_folder(), vllm_config)
    async def run():
        async for output in vllm.generate(prompt):
            print(output.outputs[0].text)
    asyncio.run(run())

Clients

OpenAI (example with Oncollama)

1. Interact with the server using the OpenAI client in python:

   from openai import OpenAI
   from oncoschema.prompt_builder import PromptBuilder # pip install londonaicentre-oncoschema
   
   client = OpenAI(
       base_url="http://localhost:5000/v1",
       api_key="blank" 
   )
   
   response = client.chat.completions.create(
       model="oncollama3betav01",
       messages=[
           {"role": "system", "content": PromptBuilder().build_main_prompt()},
           {"role": "user", "content": "Diagnosis 01/01/26..."}
       ]
   )
   
   print(response.choices[0].message.content)
   

License

This project uses a proprietary license (see LICENSE).