nvidia-eval-commons 0.1.0

No project description provided

nvidia-eval-commons

NVIDIA Eval Commons is a set of dataclasses and tools used to define the configuration for the evaluation run.

Installation

pip install nvidia-eval-commons

Usage

from nvidia_eval_commons import EvaluationConfig, EvaluationTarget, ApiEndpoint, EndpointType, ConfigParams

config = EvaluationConfig(
    type="sample_eval",
    params=ConfigParams(
        task="sample_task",
        temperature=1e-5,
        top_p=1e-5,
        extra={"n_samples": 5}
    ),
    output_dir=".",
    supported_endpoint_types=[EndpointType.CHAT, EndpointType.COMPLETIONS],
)

target = EvaluationTarget(
    api_endpoint=ApiEndpoint(
        model_id="sample_model",
        type=EndpointType.CHAT,
        url="http://sample_model:8000",
    )
)

License

Apache License, Version 2.0