views_pipeline_core 3.3.4

A modular Python framework for end‑to‑end conflict forecasting: data ingestion, transformation, drift monitoring, model and ensemble management, evaluation, reconciliation, mapping, reporting, packaging, and artifact governance.

Acknowledgements

Table of Contents

1. Conceptual Overview
2. High‑Level Architecture
3. Core Pipeline Stages
4. Managers (Orchestration Layer)
5. Modules (Functional Layer)
6. Data Layer & Querysets
7. Evaluation & Metrics
8. Reconciliation (Hierarchical Consistency)
9. Reporting & Mapping
10. CLI & Argument System
11. Configuration & Partitioning
12. Package Management
13. Logging & Monitoring
14. Development Workflow
15. Quick Start
16. FAQ

1. Conceptual Overview

The pipeline transforms raw geo‑temporal data into validated, reconciled, and documented forecasts. Key features include:

• Deterministic data preparation (queryset-driven, drift-checked)
• Strict naming & artifact conventions
• Partition-aware evaluation (calibration/validation/forecasting)
• Multi-model ensembling & hierarchical reconciliation
• Automated HTML reporting and spatial visualization
• Reproducible configuration merging and logging
• Optional integration with Weights & Biases (WandB) and prediction store

2. High‑Level Architecture

         ┌────────────────────────────────────────┐
         │            ConfigurationManager        │
         │  (deployment + hyperparameters + meta) │
         └───────────────┬────────────────────────┘
                         │
┌────────────────────────▼─────────────────────────┐
│                 ViewsDataLoader                  │
│  Queryset → Raw Fetch → Drift Check              │
│  → Partition Slice                               │
└───────────────┬──────────────────────────────────┘
                │  DataFrame (month_id, entity_id)
                ▼
      ┌─────────────────────────────────────────┐
      │   ForecastingModelManager (thin façade)  │
      │   Routes to independently testable stages│
      │                                         │
      │   ┌─────────────┐  ┌──────────────────┐ │
      │   │TrainingStage│  │ EvaluationStage  │ │
      │   └─────────────┘  └──────────────────┘ │
      │   ┌───────────────┐ ┌────────────────┐  │
      │   │ForecastingStage│ │ReportingStage │  │
      │   └───────────────┘ └────────────────┘  │
      │   ┌───────────────────────┐              │
      │   │  PredictionIOManager  │              │
      │   └───────────────────────┘              │
      └────────────┬────────────────────────────┘
                   │ Predictions
                   ▼
         ┌────────────────────────┐
         │ ReconciliationModule   │
         │ (Country ↔ Priogrid)   │
         └────────────┬───────────┘
                      │ Reconciled Predictions
                      ▼
         ┌───────────────────────────┐
         │ Reporting & Mapping       │
         │ HTML, Tables, Choropleths │
         └───────────────────────────┘

Architecture note (April 2026): ForecastingModelManager has been decomposed from a 3049-LOC monolith into a ~1960-LOC thin façade that delegates to 5 independently testable stage classes. Each stage receives a frozen context object (immutable dataclass) rather than reaching into the manager's internals. ModelPathManager has been relocated from managers/ to data/ to fix dependency inversion. All existing import paths continue to work via re-export shims. See ADR-045 for the full design rationale.

3. Core Pipeline Stages

4. Managers (Orchestration Layer)

Each stage receives an immutable frozen context object and is independently unit-testable. See ADR-045 for architecture details.

5. Modules (Functional Layer)

5.1 Intermediate Modules

6. Data Layer & Querysets

• Querysets define feature/target extraction logic + transformation chains.
• MultiIndex structure: (month_id, entity_id) for time-spatial operations.
• Data types normalized (float64 for numeric integrity).
• Partitions defined via month ranges (train/test or forecast horizon).

7. Evaluation & Metrics

Evaluation produces:

• Step-wise metrics (per forecast horizon)
• Month-wise metrics (temporal slices)
• Time-series metrics (sequence performance trajectory)

Conflict type auto-inferred from target tokens (sb / ns / os). Files named per ADR conventions (artifact/output naming).

8. Reconciliation (Hierarchical Consistency)

Ensures priogrid sums align with authoritative country totals while preserving relative spatial pattern and zero inflation. Parallelizable across countries × time × targets. Integrated into ensembles or model forecast postprocessing.

9. Reporting & Mapping

Reports embed:

• Metrics tables
• Key–value configuration summaries
• Spatial animations (Plotly)
• Artifact provenance (timestamps, versions)

10. CLI & Argument System

Dataclass-driven (ForecastingModelArgs):

• Flags: --train, --evaluate, --forecast, --report, --sweep, --prediction_store, --monthly
• Validation prevents illegal combinations (e.g., evaluate with forecasting run type).
• Monthly shortcut auto-configures production cycle.

11. Configuration & Partitioning

ConfigurationManager merges:

1. Deployment
2. Hyperparameters
3. Meta
4. Partition dictionary
5. Runtime overrides (highest priority)

Forecast partitions dynamically adjusted by override_timestep. Validation enforces structural integrity and target specification.

12. Package Management

PackageManager:

• Validates naming (organization-prefix-*)
• Creates Poetry skeleton (Python version constraint)
• Adds dependencies (including views-pipeline-core)
• Fetches latest release (tags or GitHub API)
• Runs poetry check

13. Logging & Monitoring

• YAML-driven configuration (handlers, levels, formatters).
• Dedicated model/ensemble logging directories.
• Standard separation: main log, error log.
• WandB alerts for stage transitions, failures, reconciliation completeness.

14. Development Workflow

Refer to documentation/development_guidelines.md for coding standards and docstring_guidelines.md for formatting.

15. Quick Start

1. Run build_model_scaffold.py or build_ensemble_scaffold.py found in the views-models repository.

2. Update config_maturity.py, config_hyperparameters.py, config_queryset.py, config_meta.py. (Sources scaffolded before ADR-057 carry config_deployment.py instead; either is read while the transition window is open, and the new name wins when both are present.)

3. Run calibration:

   python main.py --run_type calibration --train --evaluate --report
   

4. Run forecasting:

   python main.py --run_type forecasting --train --forecast --report
   

5. View artifacts: models/<name>/artifacts/

16. FAQ