nvidia-eval-commons

NVIDIA Eval Commons is a set of dataclasses and tools used to define the configuration for the evaluation run.

Installation

pip install nvidia-eval-commons

Note: This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

Usage

from nvidia_eval_commons.api.api_dataclasses import EvaluationConfig, EvaluationTarget, ApiEndpoint, EndpointType, ConfigParams

config = EvaluationConfig(
    type="sample_eval",
    params=ConfigParams(
        task="sample_task",
        temperature=1e-5,
        top_p=1e-5,
        extra={"n_samples": 5}
    ),
    output_dir=".",
    supported_endpoint_types=[EndpointType.CHAT, EndpointType.COMPLETIONS],
)

target = EvaluationTarget(
    api_endpoint=ApiEndpoint(
        model_id="sample_model",
        type=EndpointType.CHAT,
        url="http://sample_model:8000",
    )
)

Development

Install in editable mode with [all] extra:

pip install -e '.[all]'

This installs testing dependencies.

License

Apache License, Version 2.0