iomeval 0.0.4

Streamline evaluation evidence mapping at scale with LLMs

IOMEval

iomeval streamlines the mapping of IOM evaluation reports against strategic frameworks like the Strategic Results Framework (SRF) and Global Compact for Migration (GCM). It uses LLMs to process PDF reports, extract key sections, and tag/map them to framework components, turning dispersed, untagged evaluation documents into structured, searchable knowledge maps.

Why This Matters

UN agencies produce extensive evaluation reports and other public documents. For IOM, this body of knowledge is extensive and variegated, but putting it to practical use becomes more challenging as volume increases, particularly when documentation is stored across different repositories with no single index available.

The Challenge for IOM

IOM’s evaluation production is highly decentralized, with reports stored across multiple repositories (the IOM Evaluation Repository, IOM Library, IOM Protection Platform). Quality varies greatly—quality control processes are not applied uniformly, and variation also reflects the inherent subjectivity in evaluation approaches and interpretations. Reports also vary significantly in structure: some follow common formats with executive summaries, findings, and recommendations, while others have different structures entirely. This inconsistency makes systematic mapping challenging.

Critically, existing metadata doesn’t indicate which elements of IOM’s strategic frameworks—the Strategic Results Framework (SRF) or the Global Compact for Migration (GCM)—each report addresses. This is a major gap that limits the ability to connect evaluation evidence with key strategic frameworks.

Evidence Maps as a Solution

Evidence Maps display the extent and nature of research and evaluation available on a subject. Following the 2025 UNEG Eval Week, four primary use cases emerged: guiding future evidence generation, informing policy decisions, knowledge management, and enhancing collaboration. The maps created by iomeval serve primarily as knowledge management tools—structured repositories that make identifying relevant sources easier by organizing them against strategic framework components.

What This Enables

By tagging reports against SRF outputs, enablers, cross-cutting priorities, and GCM objectives, these maps help answer questions like: Which framework elements are well-covered by existing evaluations? Where are the knowledge gaps that should prioritize future evaluation work? Which themes have enough evidence for a dedicated synthesis report?

Key Features

• Automated PDF Processing: Download and OCR evaluation reports with proper heading hierarchy
• Intelligent Section Extraction: LLM-powered extraction of executive summaries, findings, conclusions, and recommendations
• Strategic Framework Mapping: Map report content to IOM’s SRF Enablers, Cross-Cutting Priorities, GCM Objectives, and SRF Outputs
• Checkpoint/Resume: Built-in state persistence - interrupt and resume long-running pipelines
• Granular Control: Use the full pipeline or individual components as needed

Installation

Install from PyPI:

pip install iomeval

Or install the latest development version from GitHub:

pip install git+https://github.com/franckalbinet/iomeval.git

Configuration

Core Dependencies

iomeval relies on two key libraries:

• mistocr: Powers the PDF-to-markdown conversion with intelligent OCR and heading hierarchy detection
• lisette: A thin wrapper around litellm that provides access to all major LLM providers. By default, iomeval uses Anthropic models (Haiku for debugging, Sonnet for production)

API Keys

Set your required API keys:

export ANTHROPIC_API_KEY='your-key-here'
export MISTRAL_API_KEY='your-key-here'

Since lisette supports all major LLM providers via litellm, you can alternatively configure other providers (OpenAI, Google, etc.) by setting their respective API keys.

Quick Start

First, prepare your evaluations data. Export evaluations from IOM’s evaluation repository as CSV, then convert to JSON:

from iomeval.readers import IOMRepoReader

reader = IOMRepoReader('evaluation-search-export.csv')
reader.to_json('evaluations.json')

Now process an evaluation report end-to-end:

from iomeval.readers import load_evals
from iomeval.pipeline import run_pipeline

evals = load_evals('evaluations.json')
url = "https://evaluation.iom.int/sites/g/files/tmzbdl151/files/docs/resources/Abridged%20Evaluation%20Report_%20Final_Olta%20NDOJA.pdf"

report = await run_pipeline(url, evals, 
                            pdf_dst='data/pdfs', 
                            md_dst='data/md', 
                            results_path='data/results', 
                            ocr_kwargs=dict(add_img_desc=False), 
                            model='claude-haiku-4-5')
report

The pipeline runs 7 steps: download → OCR → extract → map enablers → map CCPs → map GCM objectives → map outputs. Progress is displayed as each step completes, and state is automatically saved after each stage for checkpoint/resume capability.

Detailed Workflow

For more control over individual pipeline stages, see the module documentation:

• Loading evaluation metadata: See readers for working with IOM evaluation data
• Downloading and OCR: See downloaders and core for PDF processing
• Section extraction: See extract for extracting executive summaries, findings, conclusions, and recommendations
• Framework mapping: See mapper for mapping to SRF enablers, cross-cutting priorities, GCM objectives, and SRF outputs
• Pipeline control: See pipeline for granular control over the full pipeline and checkpoint/resume functionality

Development

iomeval is built with nbdev, which means the entire library is developed in Jupyter notebooks. The notebooks serve as both documentation and source code.

Setup for development

git clone https://github.com/franckalbinet/iomeval.git
cd iomeval
pip install -e '.[dev]'

Key nbdev commands

nbdev_test          # Run tests in notebooks
nbdev_export        # Export notebooks to Python modules
nbdev_preview       # Preview documentation site
nbdev_prepare       # Export, test, and clean notebooks (run before committing)

Workflow

1. Make changes in the .ipynb notebook files
2. Run nbdev_prepare to export code and run tests
3. Commit both notebooks and exported Python files
4. Documentation is automatically generated from the notebooks

Learn more about nbdev’s literate programming approach in the nbdev documentation.

Contributing

Contributions are welcome! Please: - Follow the existing notebook structure - Run nbdev_prepare before submitting PRs