aitutor-assessmentkit 0.1.4

AITutor-AssessmentKit is the first open-source toolkit designed to evaluate the pedagogical performance of AI tutors in student mistake remediation tasks. With the growing capabilities of large language models (LLMs), this library provides a systematic approach to assess their teaching potential across multiple dimensions in educational dialogues.

{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# AITutor-AssessmentKit: An Open-Source Library to Measure Pedagogical Ability of AI Tutors in Educational Dialogues\n", "\n", "---\n", "\n", "The AITutor-AssessmentKit is the first open-source library to enable the pedagogical abilities assessment of large language model (LLM)-powered AI tutors in educational dialogues. This unified framework:\n", "\n", "- Evaluates AI tutor responses across eight comprehensive dimensions in the context of student error remediation tasks in mathematics.\n", "- Offers a pluggable and customizable interface for integrating models and LLM releases from the community.\n", "\n", "By providing an efficient, scalable alternative to costly and subjective human evaluations, AITutor-AssessmentKit facilitates on-the-fly assessment of AI tutors.\n", "\n", "---\n", "\n", "## 📥 Installation\n", "\n", "To install AITutor-AssessmentKit with pip, type:\n", "\n", "bash\n", "pip install aitutor-assessmentkit\n", "\n", "\n", "---\n", "\n", "## 📚 Overview of the AITutor-AssessmentKit\n", "\n", "The library comprises three modular components:\n", "\n", "1. autoeval: For automated evaluation.\n", "2. llmeval: For LLM-based evaluation.\n", "3. visualizer: For visualization and interpretation of evaluation scores.\n", "\n", "\n", "\n", "---\n", "\n", "## 📖 Tutorials\n", "\n", "We provide several resources to help you get started:\n", "\n", "1. Tutorial Notebook-I: Automated Evaluation with AITutor-AssessmentKit\n", "2. Tutorial Notebook-II: LLM-Based Evaluation with AITutor-AssessmentKit\n", "3. Tutorial Notebook-III: Visualizing AI Tutor Performance with AITutor-AssessmentKit\n", "\n", "For a quick overview, check out our Demo Notebook.\n", "\n", "---\n", "\n", "## 🚀 Next Steps\n", "\n", "Here are the upcoming milestones for the project:\n", "\n", "1. Create and release detailed documentation.\n", "2. Publish a longer tutorial video.\n", "3. Develop a GUI-friendly interaction mode.\n", "\n", "---\n", "\n", "## 📜 Citation\n", "\n", "If you use AITutor-AssessmentKit in your research, please cite us:\n", "\n", "bibtex\n", "@inproceedings{maurya2024aitutorassessmentkit,\n", " title={AITutor-AssessmentKit: An Open-Source Library to Measure Pedagogical Ability of AI Tutors in Educational Dialogues},\n", " author={Kaushal Kumar Maurya and Ekaterina Kochmar},\n", " year={2024}\n", "}\n", "\n", "\n", "---\n", "\n", "## 📧 Contact\n", "\n", "For any questions or support, feel free to reach out:\n", "\n", "📧 Kaushal Kumar Maurya: [e-mail]\n", "\n", "---\n" ] } ], "metadata": { "kernelspec": { "display_name": "pds_py310", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.13" } }, "nbformat": 4, "nbformat_minor": 2 }