evaluatr 0.0.3

Streamline policy evaluation workflows with AI-driven analysis and evaluation framework-agnostic processing

Evaluatr

What is Evaluatr?

Evaluatr is an AI-powered system that automates the creation of Evidence Maps - structured, visual tools that organize what we know (and don’t know) about programs, policies, and interventions. It transforms hundreds of hours of manual document review into an automated, intelligent process.

The Challenge We Solve

Traditional evidence mapping requires hundreds of staff-hours manually reviewing and tagging evaluation documents. Evaluatr automates this process while maintaining accuracy through human-AI collaboration, making evidence mapping accessible and efficient.

Key Features

Document Processing

• Multi-Format Repository Reading: Seamlessly process evaluation repositories from diverse organizations with standardized outputs
• OCR Processing: Extract text from scanned PDFs and images to ensure no valuable information is missed
• Intelligent Document Chunking: Break down documents into meaningful segments for optimal analysis

AI-Powered Analysis

• Automated Information Extraction: Extract key program details, context, and findings using advanced AI models
• Smart Document Tagging: AI-assisted categorization and labeling of evaluation reports
• Enhanced RAG Implementation: Enriched chunking strategy for more accurate retrieval and generation

Flexibility & Openness

• Framework Agnostic Design: Compatible with multiple evaluation frameworks including IOM Strategic Results Framework and Global Compact on Migration
• Open Training Data: Contribute to and benefit from community-curated training datasets

️ Installation

From PyPI (Recommended)

pip install evaluatr

From GitHub

pip install git+https://github.com/franckalbinet/evaluatr.git

Development Installation

# Clone the repository
git clone https://github.com/franckalbinet/evaluatr.git
cd evaluatr

# Install in development mode
pip install -e .

# Make changes in nbs/ directory, then compile:
nbdev_prepare

Quick Start

Reading an IOM Evaluation Repository

from evaluatr.readers import IOMRepoReader

# Initialize reader with your Excel file
reader = IOMRepoReader('files/test/eval_repo_iom.xlsx')

# Process the repository
evaluations = reader()

# Each evaluation is a standardized dictionary
for eval in evaluations[:3]:  # Show first 3
    print(f"ID: {eval['id']}")
    print(f"Title: {eval['meta']['Title']}")
    print(f"Documents: {len(eval['docs'])}")
    print("---")

ID: 1a57974ab89d7280988aa6b706147ce1
Title: EX-POST EVALUATION OF THE PROJECT:  NIGERIA: STRENGTHENING REINTEGRATION FOR RETURNEES (SRARP)  - PHASE II
Documents: 2
---
ID: c660e774d14854e20dc74457712b50ec
Title: FINAL EVALUATION OF THE PROJECT: STRENGTHEN BORDER MANAGEMENT AND SECURITY IN MALI AND NIGER THROUGH CAPACITY BUILDING OF BORDER AUTHORITIES AND ENHANCED DIALOGUE WITH BORDER COMMUNITIES
Documents: 2
---
ID: 2cae361c6779b561af07200e3d4e4051
Title: Final Evaluation of the project "SUPPORTING THE IMPLEMENTATION OF AN E RESIDENCE PLATFORM IN CABO VERDE"
Documents: 2
---

Exporting it to JSON:

reader.to_json('processed_evaluations.json')

Downloading evaluation documents

from evaluatr.downloaders import download_docs
from pathlib import Path

fname = 'files/test/evaluations.json'
base_dir = Path("files/test/pdf_library")
download_docs(fname, base_dir=base_dir, n_workers=0, overwrite=True)

(#24) ['Downloaded Internal%20Evaluation_NG20P0516_MAY_2023_FINAL_Abderrahim%20EL%20MOULAT.pdf','Downloaded RR0163_Evaluation%20Brief_MAY_%202023_Abderrahim%20EL%20MOULAT.pdf','Downloaded IB0238_Evaluation%20Brief_FEB_%202023_Abderrahim%20EL%20MOULAT.pdf','Downloaded Internal%20Evaluation_IB0238__FEB_2023_FINAL%20RE_Abderrahim%20EL%20MOULAT.pdf','Downloaded IB0053_Evaluation%20Brief_SEP_%202022_Abderrahim%20EL%20MOULAT.pdf','Downloaded Internal%20Evaluation_IB0053_OCT_2022_FINAL_Abderrahim%20EL%20MOULAT_0.pdf','Downloaded Internal%20Evaluation_NC0030_JUNE_2022_FINAL_Abderrahim%20EL%20MOULAT_0.pdf','Downloaded NC0030_Evaluation%20Brief_June%202022_Abderrahim%20EL%20MOULAT.pdf','Downloaded CD0015_Evaluation%20Brief_May%202022_Abderrahim%20EL%20MOULAT.pdf','Downloaded Projet%20CD0015_Final%20Evaluation%20Report_May_202_Abderrahim%20EL%20MOULAT.pdf','Downloaded Internal%20Evaluation_Retour%20Vert_JUL_2021_Fina_Abderrahim%20EL%20MOULAT.pdf','Downloaded NC0012_Evaluation%20Brief_JUL%202021_Abderrahim%20EL%20MOULAT.pdf','Downloaded Nigeria%20GIZ%20Internal%20Evaluation_JANUARY_2021__Abderrahim%20EL%20MOULAT.pdf','Downloaded Nigeria%20GIZ%20Project_Evaluation%20Brief_JAN%202021_Abderrahim%20EL%20MOULAT_0.pdf','Downloaded Evaluation%20Brief_ARCO_Shiraz%20JERBI.pdF','Downloaded Final%20evaluation%20report_ARCO_Shiraz%20JERBI_1.pdf','Downloaded Management%20Response%20Matrix_ARCO_Shiraz%20JERBI.pdf','Downloaded IOM%20MANAGEMENT%20RESPONSE%20MATRIX.pdf','Downloaded IOM%20Niger%20-%20MIRAA%20III%20-%20Final%20Evaluation%20Report%20%28003%29.pdf','Downloaded CE.0369%20-%20IDEE%20-%20ANNEXE%201%20-%20Rapport%20Recherche_Joanie%20DUROCHER_0.pdf'...]

Documentation

• Full Documentation: GitHub Pages
• API Reference: Available in the documentation
• Examples: See the nbs/ directory for Jupyter notebooks

Contributing

We welcome contributions! Here’s how you can help:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes in the nbs/ directory
4. Compile with nbdev_prepare
5. Commit your changes (git commit -m 'Add amazing feature')
6. Push to the branch (git push origin feature/amazing-feature)
7. Open a Pull Request

Development Setup

# Install development dependencies
pip install -e .

# Make changes in nbs/ directory
# ...

# Compile changes to evalstack package
nbdev_prepare

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• Built with nbdev for literate programming
• Uses pandas for data processing
• Powered by rich for beautiful terminal output

Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Email: [Your email here]