paramount 0.3.2

Business Evals for LLMs

paramount

Business Evals for LLMs

Getting Started

Install Packages

Server

To enter Python Virtual Env

source venv/bin/activate # and for exiting, deactivate

pip install paramount # or python3 -m pip install paramount

Client

cd client
pnpm i

Running

Server

This will run the server and the server will serve the client as well at localhost:9001

gunicorn --bind :9001 --workers 1 --threads 8 --timeout 0 paramount.server.wsgi:app # or make run-server

Health Check: localhost:9001/health

Client

For better development experience, make sure uncomment the API_URL constant inside services.ts and add it to every endpoint. With this, you can run the client separately and gain the advantage of HMR.

cd client
pnpm dev  # or make run-client

Building the client

Make sure before pushing/deploying the latest changes, run this command to get the latest build of the client. Also, after build command, don't forget the add the __init__.py file to both dist & dist/assets folders.

cd client
pnpm run build

Environment Variables

There are two ways of accessing environment variables depending on the case of whether you'll use the client side with letting the server serve the static files or running both separately.

Using .env file

You need to also create some of the environment files with using VITE_ prefix for the client. You can access the variables on client using import.meta.env.<envVarName>

FUNCTION_API_BASE_URL="http://localhost:9000"
PARAMOUNT_DB_TYPE="postgres"
PARAMOUNT_POSTGRES_CONNECTION_STRING=
PARAMOUNT_IDENTIFIER_COLNAME="input_args__company_uuid"
PARAMOUNT_META_COLS="['recorded_at']"
PARAMOUNT_INPUT_COLS="['args__message_history', 'args__new_question']"
PARAMOUNT_OUTPUT_COLS="['1_answer', '1_based_on']"
PARAMOUNT_IS_LIVE="TRUE"
PARAMOUNT_API_ENDPOINT="http://localhost:9001"
PARAMOUNT_CONFIG_FILE=
APP_ENV=development

VITE_API_ENDPOINT="http://localhost:9001"
VITE_META_COLS="["recorded_at"]"
VITE_INPUT_COLS="["args__message_history", "args__new_question"]"
VITE_OUTPUT_COLS="["1_answer", "1_based_on"]"

Using .toml file

You can access the variables on client like this;

NOTE: .toml file should be accessed from either on the root path or from the PARAMOUNT_CONFIG_FILE path.

import paramountConfig from "./paramount.toml";

console.log(paramountConfig);

Here is the example toml file

Also, you can get the .toml configuration for the UI using the /api/config endpoint as well. Accessing these config files can be done by using the functions inside utils.ts depending on your case.

Docker

By using just Dockerfile.server, you can containerize and deploy the whole package (including the client).

For running/deploying the server & client separately;

Client

docker build -t paramount-client -f Dockerfile.client . # or make docker-build-client
docker run -dp 3002:3002 paramount-client # or make docker-run-client

Server

docker build -t paramount-server -f Dockerfile.server . # or make docker-build-server
docker run -dp 9001:9001 paramount-server # or make docker-run-server

TODOs

• Fix env variables for the client (.toml fix)
• Refactor Dockerfile for the client (Also file names have been changed for both client & server)
• Allow Flask to serve the static files of the client
• Check the .env file access from Docker, add a prod check for the absolute path

pypi upload procedure

prerequisite

pip install wheel twine

env vars for auth

export TWINE_USERNAME=__token__
export TWINE_PASSWORD=<your_pypi_api_token>

build

python setup.py sdist bdist_wheel

upload

twine upload dist/*