probLM-solver

A Python library enabling researchers to prototype and develop sampling methods in large language models (LLMs) based on internal probabilities. Included are utilities for writing complex sampling functions in Python and running local models (with llama.cpp as a backend) that use them.

This library ensures researchers can sample token-by-token, analyse logits, and assess the probability of given outputs.

An R wrapper around this library is available as problmsolver.

This library is currently in active development, and looking for feature requests. Breaking changes will be kept to a minimum, and loudly declared in the changelog where they are unavoidable.

Usage

Bare Metal

To install this package as a library, please run:

pip install problm-solver

to install the latest release in your desired virtual environment.

For the latest main branch, please clone the repository and install via,

python3 -m pip install /path/to/probLM-solver

If you need the project's working environment as a starting point, see the instructions below:

This project is managed with uv, ensure you have Python, pip, and uv installed. To create and activate the virtual environment, run:

uv sync --python 3.13 --no-dev
source .venv/bin/activate

Conda

An environment.yml is provided for conda users (Miniforge, Miniconda, or Anaconda). To create and activate the environment, run:

conda env create -f environment.yml
conda activate problm-solver

If you have an NVIDIA GPU and want hardware-accelerated inference, reinstall llama-cpp-python with CUDA support after activating the environment:

CMAKE_ARGS="-DGGML_CUDA=on" pip install "llama-cpp-python>=0.3.23" --force-reinstall

On the HEC, and other HPC systems which use Nvidia L40(S) and H200 GPUs, oftentimes the login node won't have native GPUs, so use the command:

CMAKE_ARGS="-DGGML_CUDA=on -DGGML_NATIVE=off -DCMAKE_CUDA_ARCHITECTURES=89;90" pip install --force-reinstall --no-cache-dir "llama-cpp-python>=0.3.23"

For complete and custom installation of llama-cpp-python, see llama.cpp build docs.

Docker

The CLI itself can be run in a docker container for testing the program.

To serve this app, run:

docker compose --profile app run --build app

Within the Dev Container this is equivalent to:

poe serve

Contributing