git clone https://github.com/llm-bg/Tucan-Eval.git
cd Tucan-Eval && pip install -e .

Core Arguments

--model, -m MODEL           # Model name/path (required for evaluation)
                           # Format: model_name[,param=value,...]
                           # Example: INSAIT-Institute/BgGPT-Gemma-2-2.6B-IT-v1.0,dtype=float16
--device DEVICE             # Device: auto, cpu, cuda, cuda:0, etc. (default: auto)
--batch_size SIZE           # Batch size for inference (default: 1)

Generation Parameters

--gen_kwargs PARAMS         # Override default generation parameters
                           # Format: comma-separated key=value pairs
                           # Default values are optimized for BgGPT models
                           # Example: temperature=0.1,top_k=25,max_new_tokens=2048,eos_token_id=[1,107]

Model Parameters (via --model)

# Available model parameters:
dtype=bfloat16              # Data type (bfloat16, float16, float32)
load_in_4bit=true          # Enable 4-bit quantization
attn_implementation=eager   # Attention implementation (eager for Gemma models)

Data Arguments

--samples, -s PATH          # Path to evaluation samples
--source_type TYPE          # auto, local, hf_dataset, hf_file (default: auto)
--split SPLIT              # Dataset split (train, test, validation)
--subset SUBSET            # Dataset subset/configuration

Authentication

--hf_token TOKEN           # HuggingFace token for private models
--openai_api_key KEY       # OpenAI API key

Output & Debugging

--output_path, -o PATH     # Output directory or file (default: current directory)
--log_samples              # Log detailed sample info for debugging
--verbose, -v              # Enable verbose logging to debug.log

Utility Commands

--preview_dataset          # Preview dataset structure without evaluation
--list_files REPO          # List available files in HF repository

Advanced Options

--limit N                  # Limit number of samples to evaluate
--system_prompt TEXT       # Custom system prompt template
--tool_call_format TAGS    # Tool call format as start_tag,end_tag (default: ```tool_call,```)

# Text Customization (optimized for Bulgarian by default)
--functions_header TEXT    # Header for functions section (default: "## Налични функции:")
--user_query_header TEXT   # Header for user query section (default: "## Потребителска заявка:")
--user_prefix TEXT         # Prefix for user messages (default: "Потребител:")
--default_system_prompt TEXT  # Default system prompt (default: Bulgarian text)
--function_system_prompt_template TEXT  # Jinja2 template for function system prompt

Tucan generates comprehensive JSON reports optimized for function-calling evaluation:

{
  "model_info": {
    "model_name": "llm-bg/Tucan-9B-v1.0",
    "model_type": "huggingface",
    "generation_params": {...}
  },
  "evaluation_summary": {
    "total": 100,
    "correct": 85,
    "accuracy": 0.85,
    "by_scenario_type": {...},
    "error_distribution": {...}
  },
  "detailed_results": [...],
  "metadata": {...}
}

The evaluation automatically prints a summary to console:

📊 EVALUATION SUMMARY
===============================================================================
🎯 Overall Accuracy: 85.00% (85/100)

📈 Accuracy by Scenario Type:
function_call_required               90.00%     (45/50)
irrelevant_question_with_functions   80.00%     (40/50)

📉 Error Distribution:
WRONG_PARAMETERS                     8     (53.33% of errors)
NO_CALL_WHEN_EXPECTED               4     (26.67% of errors)
UNEXPECTED_CALL                      3     (20.00% of errors)
===============================================================================

Tucan evaluation framework supports full customization of prompts and headers for different languages and use cases:

🎯 Key Features:

• Configurable Headers: Customize section headers for functions and user queries
• Multi-language Prompts: Switch between Bulgarian, English, or any language
• Template System: Use Jinja2 templates for complex prompt structures
• User Prefix Control: Customize how user messages are prefixed
• Default Behavior: Optimized for Bulgarian Tucan models out-of-the-box

📝 Default (Bulgarian)

# Uses Bulgarian headers and prompts (default)
tucan --model llm-bg/Tucan-9B-v1.0 --samples test.json

🇺🇸 English Evaluation

tucan --model your-model \
      --functions_header "## Available Functions:" \
      --user_query_header "## User Query:" \
      --user_prefix "User:" \
      --default_system_prompt "You are a helpful AI assistant that provides useful and accurate responses." \
      --samples test.json

Multi-Language Customization

# English evaluation with custom headers
tucan --model llm-bg/Tucan-9B-v1.0 \
      --samples test_dataset.json \
      --functions_header "## Available Functions:" \
      --user_query_header "## User Query:" \
      --user_prefix "User:" \
      --default_system_prompt "You are a helpful AI assistant that provides useful and accurate responses." \
      --device cuda

# Bulgarian function calling (default behavior)
tucan --model llm-bg/Tucan-9B-v1.0 \
      --samples bulgarian_function_calling_dataset.json \
      --system_prompt "Ти си полезен AI assistent, който може да извиква функции..." \
      --tool_call_format '```tool_call,```' \
      --device cuda \
      --verbose

# Custom system prompt template for functions
tucan --model llm-bg/Tucan-9B-v1.0 \
      --function_system_prompt_template "You are an AI assistant with access to functions. Use {{ tool_call_start_tag }} and {{ tool_call_end_tag }} for function calls." \
      --functions_header "🔧 Functions:" \
      --user_query_header "❓ Query:" \
      --samples test.json

Language-Specific Evaluations

# Compare Bulgarian vs English prompting on the same model
tucan --model llm-bg/Tucan-9B-v1.0 \
      --samples test.json \
      --output_path results/tucan_bulgarian.json

tucan --model llm-bg/Tucan-9B-v1.0 \
      --samples test.json \
      --functions_header "## Available Functions:" \
      --user_query_header "## User Query:" \
      --user_prefix "User:" \
      --default_system_prompt "You are a helpful AI assistant." \
      --output_path results/tucan_english.json

Hyperparameter Optimization for Tucan Models

# Test different generation parameters optimized for Tucan
tucan --model llm-bg/Tucan-9B-v1.0 \
      --gen_kwargs temperature=0.1,top_k=25,repetition_penalty=1.1 \
      --samples test.json --output_path results/tucan_config1.json

tucan --model llm-bg/Tucan-9B-v1.0 \
      --gen_kwargs temperature=0.3,top_k=50,repetition_penalty=1.2 \
      --samples test.json --output_path results/tucan_config2.json