probLM-solver

A python library and CLI app built on top of llama.cpp which can generate and modify language model outputs based on internal probabilities. Included are utilities for generating large numbers of LLM outputs for performing statistical analysis, as well as outputting and modifying token-by-token conditional probabilities for a given output.

Of particular importance is the ability to define sampling functions for token generation, based off of the answer's power distribution, via iteratively sampling branches of future tokens.

This library is currently in active development, so expect large changes.

Usage

Bare Metal

This project is managed with uv, ensure you have Python, pip, and uv installed. To create and activate the virtual environment, run:

uv sync --python 3.13 --no-dev
source .venv/bin/activate

To install this package as a library, if you have cloned the repository, run:

python3 -m pip install /path/to/repository

else, if you only desire the library, run:

pip install git+https://github.com/muse-writes/probLM-solver

Conda

An environment.yml is provided for conda users (Miniforge, Miniconda, or Anaconda). To create and activate the environment, run:

conda env create -f environment.yml
conda activate problm-solver

If you have an NVIDIA GPU and want hardware-accelerated inference, reinstall llama-cpp-python with CUDA support after activating the environment:

CMAKE_ARGS="-DGGML_CUDA=on" pip install "llama-cpp-python>=0.3.23" --force-reinstall

On the HEC, and other HPC systems which use Nvidia L40(S) and H200 GPUs, oftentimes the login node won't have native GPUs, so use the command:

CMAKE_ARGS="-DGGML_CUDA=on -DGGML_NATIVE=off -DCMAKE_CUDA_ARCHITECTURES=89;90 pip install --force-reinstall --no-cache-dir "llama-cpp-python>=0.3.23"

For complete and custom installation of llama-cpp-python, see llama.cpp build docs.

Docker

The CLI itself can be run in a docker container for testing the program.

To serve this app, run:

docker-compose run app --build

Within the Dev Container this is equivalent to:

poe serve

Contributing