nvidia-genai-perf-eval 26.1

GenAI Perf - packaged by NVIDIA Evals Factory

NVIDIA NeMo Evaluator

The goal of NVIDIA NeMo Evaluator is to advance and refine state-of-the-art methodologies for model evaluation, and deliver them as modular evaluation packages (evaluation containers and pip wheels) that teams can use as standardized building blocks.

Quick start guide

NVIDIA NeMo Evaluator provide you with evaluation clients, that are specifically built to evaluate model endpoints using our Standard API.

Launching an evaluation for an LLM

1. Install nemo-evaluator

   pip install nemo-evaluator
   

2. Install the package

   pip install nvidia-genai-perf-eval --index-url https://gitlab-master.nvidia.com/api/v4/projects/153108/packages/pypi/simple    
   

3. (Optional) Set a token to your API endpoint if it's protected

   export MY_API_KEY="your_api_key_here"
   

4. List the available evaluations:

   $ nemo-evaluator ls
     Available tasks:
     * genai_perf_generation (in genai_perf_eval)
     * genai_perf_summarization (in genai_perf_eval)
   ...
   

5. Run the evaluation. Remember to set the tokenizer:

   nemo-evaluator run_eval \
      --eval_type genai_perf_generation \
      --model_id meta/llama-3.1-8b-instruct \
      --model_url https://integrate.api.nvidia.com/v1/chat/completions \
      --model_type chat \
      --api_key_name MY_API_KEY \
      --output_dir /workspace/results \
      --overrides="config.params.extra.tokenizer=meta-llama/Llama-3.1-70B-Instruct,target.api_endpoint.stream=true"
   

   It is recommended to turn the streaming on if your endpoint supports it (target.api_endpoint.stream=true). Otherwise only a subset of metrics can be calculated.

   If your tokenizer comes from a gated repofitory (like in the example above), you will need to set the HF_TOKEN to authenticate before running the evaluation:

   export HF_TOKEN=...
   

6. Gather the results

   cat /workspace/results/results.yml
   

Command-Line Tool

Each package comes pre-installed with a set of command-line tools, designed to simplify the execution of evaluation tasks. Below are the available commands and their usage for the genai_perf_eval:

Commands

1. List Evaluation Types

nemo-evaluator ls

Displays the evaluation types available within the harness.

2. Run an evaluation

The nemo-evaluator run_eval command executes the evaluation process. Below are the flags and their descriptions:

Required flags

• --eval_type <string> The type of evaluation to perform
• --model_id <string> The name or identifier of the model to evaluate.
• --model_url <url> The API endpoint where the model is accessible.
• --model_type <string> The type of the model to evaluate, currently either "chat", "completions", or "vlm".
• --output_dir <directory> The directory to use as the working directory for the evaluation. The results, including the results.yml output file, will be saved here. Make sure to use the absolute path.
• --overrides="config.params.extra.tokenizer=<tokenizer>" The name of the HuggingFace tokenizer used for evaluation (e.g. meta-llama/Llama-3.1-70B-Instruct). If it comes from a gated repository, you will need to export HF_TOKEN=... to authenticate.

Optional flags

• --api_key_name <string> The name of the environment variable that stores the Bearer token for the API, if authentication is required.
• --run_config <path> Specifies the path to a YAML file containing the evaluation definition.

Example

export HF_TOKEN=...

nemo-evaluator run_eval \
    --eval_type genai_perf_generation \
    --model_id my_model \
    --model_type chat \
    --model_url http://localhost:8000/v1/chat/completions \
    --output_dir /workspace/evaluation_results \
    --overrides="config.params.extra.tokenizer=meta-llama/Llama-3.1-70B-Instruct,target.api_endpoint.stream=true"

If the model API requires authentication, set the API key in an environment variable and reference it using the --api_key_name flag:

export MY_API_KEY="your_api_key_here"
export HF_TOKEN=...

nemo-evaluator run_eval \
    --eval_type genai_perf_generation \
    --model_id my_model \
    --model_type chat \
    --model_url http://localhost:8000/v1/chat/completions \
    --api_key_name MY_API_KEY \
    --output_dir /workspace/evaluation_results \
    --overrides="config.params.extra.tokenizer=meta-llama/Llama-3.1-70B-Instruct,target.api_endpoint.stream=true"

Configuring evaluations via YAML

Evaluations in NVIDIA NeMo Evaluator are configured using YAML files that define the parameters and settings required for the evaluation process. These configuration files follow a standard API which ensures consistency across evaluations.

Example of a YAML config:

config:
  type: genai_perf_generation
  params:
    parallelism: 50
    limit_samples: 20
    extra:
        tokenizer: microsoft/Phi-4-mini-instruct
target:
  api_endpoint:
    model_id: microsoft/phi-4-mini-instruct
    type: chat
    url: https://integrate.api.nvidia.com/v1/chat/completions
    api_key_name: NVIDIA_API_KEY
    stream: true

The priority of overrides is as follows:

1. command line arguments
2. user config (as seen above)
3. task defaults (defined per task type)
4. framework defaults

--dry_run option allows you to print the final run configuration and command without executing the evaluation.

Example:

nemo-evaluator run_eval \
    --eval_type genai_perf_generation \
    --model_id my_model \
    --model_type chat \
    --model_url http://localhost:8000/v1/chat/completions \
    --output_dir /workspace/evaluation_results \
    --overrides="config.params.extra.tokenizer=meta-llama/Llama-3.1-70B-Instruct" \
    --dry_run

Output:

Rendered config:

command: genai_perf_eval --model_id {{target.api_endpoint.model_id}} --url {{target.api_endpoint.url}}  {%
  if target.api_endpoint.api_key_name is not none %}--api-key {{target.api_endpoint.api_key_name}}
  {% endif %} --concurrencies {{config.params.parallelism}} --isl {{config.params.extra.isl}}
  --osl {{config.params.extra.osl}}  --tokenizer {{config.params.extra.tokenizer}}
  --endpoint-type {{target.api_endpoint.type}} --artifact-dir {{config.output_dir}}
  {% if target.api_endpoint.stream %}--streaming {% endif %}{% if config.params.extra.warmup
  %}--warmup{% endif %}
framework_name: genai_perf_eval
pkg_name: genai_perf_eval
config:
  output_dir: /workspace/evaluation_results
  params:
    limit_samples: null
    max_new_tokens: null
    max_retries: null
    parallelism: 1
    task: genai_perf_generation
    temperature: null
    request_timeout: null
    top_p: null
    extra:
      tokenizer: meta-llama/Llama-3.1-70B-Instruct
      warmup: true
      isl: 500
      osl: 5000
  supported_endpoint_types:
  - chat
  - completions
  type: genai_perf_generation
target:
  api_endpoint:
    api_key_name: null
    model_id: my_model
    stream: null
    type: chat
    url: http://localhost:8000/v1/chat/completions


Rendered command:

genai_perf_eval --model_id my_model --url http://localhost:8000/v1/chat/completions   --concurrencies 1 --isl 500 --osl 5000  --tokenizer meta-llama/Llama-3.1-70B-Instruct --endpoint-type chat --artifact-dir /workspace/evaluation_results --warmup

Customizing your evaluation

Running GenAI-Perf tasks with different values of ISL and OSL parameters

By default, nemo-evaluator supports two tasks: genai_perf_summarization and genai_perf_generation. The genai_perf_summarization task uses an input sequence length (isl) of 5000 tokens and an output sequence length (osl) of 500 tokens, testing how your endpoint handles tasks with long input and shorter output (e.g. summarizing a long text). The genai_perf_generation task uses an isl of 500 tokens and an osl of 5000 tokens, testing how your endpoint handles tasks with short input and long output (e.g. generating a long text based on a short prompt).

You can run an evaluation with your own set of parameters as follows:

1. Using the command-line tool:

   nemo-evaluator run_eval \
       --eval_type genai_perf_generation \
       --model_id my_model \
       --model_type chat \
       --model_url http://localhost:8000/v1/chat/completions \
       --api_key_name MY_API_KEY \
       --output_dir /workspace/evaluation_results \
       --overrides="config.params.extra.tokenizer=meta-llama/Llama-3.1-70B-Instruct,config.params.extra.isl=<your_isl_value>,config.params.extra.osl=<your_osl_value>"
   

2. Using a config file:

   config:
     type: genai_perf_generation
     params:
       parallelism: 1
       extra:
         isl: <your_isl_value>
         osl: <your_osl_value>
         tokenizer: meta-llama/Llama-3.1-70B-Instruct
   target:
     api_endpoint:
       url: http://localhost:8000/v1/chat/completions
       model_id: my_model
       type: chat
   

where <your_isl_value> and <your_osl_value> are your selected values of isl and osl parameters, respectively. This allows you to test other common use-cases, e.g. for testing text generation speed you can set isl=500 and osl=5000.

Testing your endpoint with concurrent requests

By default, nemo-evaluator sends a single request at a time to your endpoint. However, you can test how your endpoint handles traffic by setting the parallelism parameter to a higher value, e.g.:

1. Using the command-line tool:

   nemo-evaluator run_eval \
       --eval_type genai_perf_generation \
       --model_id my_model \
       --model_type chat \
       --model_url http://localhost:8000/v1/chat/completions \
       --api_key_name MY_API_KEY \
       --output_dir /workspace/evaluation_results \
       --overrides="config.params.extra.tokenizer=meta-llama/Llama-3.1-70B-Instruct,config.params.parallelism=<num_requests>"
   

2. Using a config file:

   config:
     type: genai_perf_generation
     params:
       parallelism: <num_requests>
       extra:
         tokenizer: meta-llama/Llama-3.1-70B-Instruct
   target:
     api_endpoint:
       url: http://localhost:8000/v1/chat/completions
       model_id: my_model
       type: chat
   

where <num_requests> is the number of concurrent requests send to the model.

FAQ

Deploying a model as an endpoint

NVIDIA NeMo Evaluator utilize a client-server communication architecture to interact with the model. As a prerequisite, the model must be deployed as an endpoint with a NIM-compatible API.

Users have the flexibility to deploy their model using their own infrastructure and tooling.

Servers with APIs that conform to the OpenAI/NIM API standard are expected to work seamlessly out of the box.