educhateval 0.1.4

A pipeline and package to implement and evaluate LLM chat bot tutors in education.

🧠 EduChatEval

A structured pipeline and Python package for evaluating interactive LLM tutor systems in educational settings.

---

🚀 Overview

This repository supports my master’s thesis, exploring how large language models can simulate and support human-like educational dialogues.

The package is designed to:

• Simulate student–tutor dialogues with role-based prompts if no real data is available
• Integrate local + open-source models (e.g., LM Studio, Hugging Face)
• Log interactions (JSON/CSV) for analysis
• Provide a framework for classification, evaluation, and fine-tuning

Here’s a rough overview of the system architecture:

---

⚙️ Installation

pip install educhateval

🤗 Integration

• 🦙 LM Studio (local LLM inference)
• 🖍️ Outlines
• 🤗 Transformers
• 🧪 Optuna for experimental tuning

📖 Documentation

Documentation	Description
📚 User Guide	Instructions on how to run simulations and analyze dialogue logs
💡 Prompt Templates	Overview of system prompts, role behaviors, and instructional strategies
🧠 API References	Full reference for the educhateval API: classes, methods, and usage
🤔 About	Learn more about the thesis project, context, and contributors

🫶🏼 Acknowdledgement

TBA

📬 Contact

Made by Laura Wulff Paaby
Feel free to reach out via:

• 🌐 LinkedIn
• 📧 [e-mail]
• 🐙 GitHub

---

Complete overview:

├── data/                                  
│   ├── generated_dialogue_data/           # Generated dialogue samples
│   ├── generated_tuning_data/             # Generated framework data for fine-tuning 
│   ├── logged_dialogue_data/              # Logged real dialogue data
│   ├── Final_output/                      # Final classified data 
│
├── Models/                                # Folder for trained models and checkpoints (ignored)
│
├── src/educhateval/                       # Main source code for all components
│   ├── chat_ui.py                         # CLI interface for wrapping interactions
│   ├── descriptive_results/               # Scripts and tools for result analysis
│   ├── dialogue_classification/           # Tools and models for dialogue classification
│   ├── dialogue_generation/               
│   │   ├── agents/                        # Agent definitions and role behaviors
│   │   ├── models/                        # Model classes and loading mechanisms
│   │   ├── txt_llm_inputs/               # System prompts and structured inputs for LLMs
│   │   ├── chat_instructions.py          # System prompt templates and role definitions
│   │   ├── chat_model_interface.py       # Interface layer for model communication
│   │   ├── chat.py                       # Main script for orchestrating chat logic
│   │   └── simulate_dialogue.py          # Script to simulate full dialogues between agents
│   ├── framework_generation/            
│   │   ├── outline_prompts/              # Prompt templates for outlines
│   │   ├── outline_synth_LMSRIPT.py      # Synthetic outline generation pipeline
│   │   └── train_tinylabel_classifier.py # Training classifier on manually made true data
│
├── .python-version                       # Python version file for (Poetry)
├── poetry.lock                           # Locked dependency versions (Poetry)
├── pyproject.toml                        # Main project config and dependencies