helmet 1.1.0

No project description provided

HELMET: Human Evaluated large Language Model Explainability Tool

Contents

• Installation helmet
• Configuration Details
• Features
• Install Helmet from source
• Deploy helmet-platform (local)
• License

Pypi Installation

pip install helmet

Overview

This package exists of two parts;

1. A python package; helmet, which you can install in your Jupyter Notebook/Sagemaker/Colab
2. A webapp: helmet-platform, which deploys an API to save al the runs & projects and interacts with the frontend. A frontend should also be deployed.

Configuration files

Platform configuration

project_setup = {
    # This should point to the NodeJS API
    platform_url: "localhost:4000"
    project_id: "<ID>"
}

Model configuration

model_checkpoint = "meta-llama/Meta-Llama-3-8B"
model_setup = {
    "checkpoint": model_checkpoint,
    # This can be enc/enc-dec/dec
    "model_type": "dec",
    # This should specify where the embeddings are stored
    "embeddings": "model.embed_tokens",
}

Run configuration

run_config = {
    # "cuda" & "cpu" are currently supported
    "device": device,
}

Load/create project

Creating a project can be done by current the following python code in your jupyter notebook.

project_id = helmet.get_or_create_project("<platform url>", "<project name>", "text_generation")

This will give you back the ID of the project, that you can then use to load the model.

After you have configured the model, platform & device, you can start loading the model like this:

model = helmet.from_pretrained(project_setup, model_setup, run_config)

Features

• Load any causal model from Huggingface.
• Create a project for your experiment
• Run experimental prompts

Demo

A demo can be found at https://helmet.jeroennelen.nl

Install from source

To use helmet in one of the examples perform the following steps:

1. Create venv with python -m venv .venv
2. Activate the venv with source .venv/bin/activate
3. Install HELMET from source (from git, when located in the home folder of helmet pip install -e .
4. Install jupyter notebook pip install jupyterlab

To remove:

1. deactivate
2. jupyter-kernelspec uninstall venv
3. rm -r venv

Running webapp locally

For this, please check the README in the webapp

Credits

Some inspiration has been drawn from a couple of other tools:

• Interpret-ml
• Ecco
• Phoenix
• Inseq
• Ferret

License

helmet is distributed under the terms of the MIT license.