llama-stack-provider-lmeval 0.1.2

Llama Stack Remote Eval Provider for TrustyAI LM-Eval

llama-stack-provider-lmeval

Llama Stack Remote Eval Provider for TrustyAI LM-Eval

About

This repository implements TrustyAI's LM-Eval as an out-of-tree Llama Stack remote provider.

It also includes an end-to-end instructions demonstratring how one can use LM-Eval on LLama Stack to run benchmark evaluations over DK-Bench on a deployed Phi-3-mini-4k-instruct model via OpenShift.

Use

Prerequsites

• Admin access to an OpenShift cluster with RHOAI installed
• Installation of uv
• Installation of oc cli tool
• Installation of llama stack cli tool

1. Clone this repository

   git clone https://github.com/trustyai-explainability/llama-stack-provider-lmeval.git
   

2. Set llama-stack-provider-lmeval/demo as your working directory.

   cd llama-stack-provider-lmeval/demo
   

3. Deploy microsoft/Phi-3-mini-4k-instruct on vLLM Serving Runtime

   a. Create a namespace with a name of your choice

   TEST_NS=<NAMESPACE>
   oc create ns $TEST_NS
   oc get ns $TEST_NS
   

   b. Deploy the model via vLLM

   oc apply -k resources/kustomization.yaml
   

4. Before continuing, preform a sanity check to make sure the model was sucessfully deployed

   oc get pods | grep "predictor"
   

   Expected output:

   phi-3-predictor-00002-deployment-794fb6b4b-clhj7   3/3     Running   0          5h55m
   

5. Retrive the model route

   VLLM_URL=$(oc get $(oc get ksvc -o name | grep predictor) --template={{.status.url}})
   

6. Create and activate a virtual enviornment

   uv venv .llamastack-venv
   

   source .llamastack-venv/bin/activate
   

7. Install the required libraries

   uv pip install -e .
   

8. In the run.yaml, make the following changes:

   a. Replace the remote::vllm url

   providers:
       inference:
       - provider_id: vllm-0
           provider_type: remote::vllm
           config:
           url: ${env.VLLM_URL:https://phi-3-predictor-llama-test.apps.rosa.p2i7w2k6p6w7t7e.3emk.p3.openshiftapps.com/v1/completions}
   

   b. Replace the remote::lmeval base_url and namespace

   - provider_id: lmeval-1
       provider_type: remote::lmeval
       config:
           use_k8s: True
           base_url: https://vllm-test.apps.rosa.p2i7w2k6p6w7t7e.3emk.p3.openshiftapps.com/v1/completions
           namespace: "llama-test"
   

9. Start the llama stack server in a virtual enviornment

   llama stack run run.yaml --image-type venv
   

   Expected output:

   INFO:     Application startup complete.
   INFO:     Uvicorn running on http://['::', '0.0.0.0']:8321 (Press CTRL+C to quit)
   

10. Navigate to demo.ipynb to run evaluation