views_evaluation 0.1.0

No project description provided

VIEWS Evaluation 📊

Part of the VIEWS Platform ecosystem for large-scale conflict forecasting.

📚 Table of Contents

1. Overview
2. Role in the VIEWS Pipeline
3. Features
4. Installation
5. Architecture
6. Project Structure
7. Contributing
8. License
9. Acknowledgements

---

🧠 Overview

The VIEWS Evaluation repository provides a standardized framework for assessing time-series forecasting models used in the VIEWS conflict prediction pipeline. It ensures consistent, robust, and interpretable evaluations through metrics tailored to conflict-related data, which often exhibit right-skewness and zero-inflation.

---

🌍 Role in the VIEWS Pipeline

VIEWS Evaluation ensures forecasting accuracy and model robustness as the official evaluation component of the VIEWS ecosystem.

Pipeline Integration:

1. Model Predictions →
2. Evaluation Metrics Processing →
3. Metrics Computation (via MetricsManager) →
4. Final Performance Reports

Integration with Other Repositories:

• views-pipeline-core: Supplies preprocessed data for evaluation.
• views-models: Provides trained models to be assessed.
• views-stepshifter: Evaluates time-shifted forecasting models.
• views-hydranet: Supports spatiotemporal deep learning model evaluations.

---

✨ Features

1. EvaluationMetrics

A data class for managing and storing evaluation metrics for time-series forecasting models.

🔹 Key Capabilities:

• Handles conflict-specific data distributions, including skewness and zero-inflation.
• Three evaluation schemas:
  1. Time-series-wise: Evaluates long-term forecasting behavior.
  2. Step-wise: Assesses performance at each forecasting step.
  3. Month-wise: Measures forecast accuracy on a rolling monthly basis.
• Transforms evaluation metrics into structured DataFrames for analysis.

📖 More details in the Evaluation Metrics Workshop Notes.

---

2. MetricsManager

A centralized evaluation engine for computing metrics on time-series forecasts.

🔹 Key Capabilities:

• Customizable metric lists allow for flexible evaluation.
• Ensures metric consistency by warning about unrecognized metrics.
• Implements all three evaluation schemas (time-series, step-wise, month-wise).
• Batch processing for multiple models and forecasting targets.

📖 More details in schema.MD.

---

3. Roadmap & Upcoming Features 🚧

✅ Planned Enhancements:

• Multi-target evaluation (e.g., assessing multiple dependent variables simultaneously).
• Expanding metric calculations beyond RMSLE, CRPS, and AP.
• New visualization tools for better interpretability of evaluation reports.

---

⚙️ Installation

Prerequisites

• Python >= 3.11

---

🏗 Architecture

1. Evaluation Metrics Framework

• Handles forecasting evaluation across multiple models, levels of analysis, and forecasting windows.
• Converts model outputs into standardized evaluation reports.

2. Metrics Computation Pipeline

1. Input: Predictions from models in standardized DataFrames.
2. Processing: Calculation of relevant evaluation metrics.
3. Output: Performance scores for comparison across models.

3. Error Handling & Standardization

• Ensures conformity to VIEWS evaluation standards.
• Warns about unrecognized or incorrectly formatted metrics.

---

🗂 Project Structure

views-evaluation/
├── README.md                   # Documentation
├── .github/workflows/           # CI/CD pipelines
├── tests/                       # Unit tests
├── views_evaluation/            # Main source code
│   ├── evaluation/
│   │   ├── metrics.py
│   ├── __init__.py              # Package initialization
├── .gitignore                   # Git ignore rules
├── pyproject.toml               # Poetry project file
├── poetry.lock                  # Dependency lock file

---

🤝 Contributing

We welcome contributions! Please follow the VIEWS Contribution Guidelines.

---

📜 License

This project is licensed under the LICENSE file.

---

💬 Acknowledgements

Special thanks to the VIEWS MD&D Team for their collaboration and support.