ImpactU Airflow ETL

Central repository for Apache Airflow DAGs and ETL (Extraction, Transformation, and Loading) processes for the ImpactU project. This package includes the full logic for data orchestration, source extraction, and data processing.

🚀 Description

This project orchestrates data collection from various scientific and academic sources, its processing using the Kahi tool, and its subsequent loading into query systems such as MongoDB and Elasticsearch.

� Installation

You can install the package directly from PyPI:

pip install impactu_airflow

Or for development:

git clone https://github.com/colav/impactu_airflow.git
cd impactu_airflow
pip install -e .

�📂 Project Structure

The repository is organized by data lifecycle stages and Airflow components:

• dags/: Apache Airflow DAG definitions.
• extract/: Extraction logic for sources like OpenAlex, ORCID, ROR, etc.
• transform/: Transformation and normalization processes (Kahi).
• load/: Loading scripts to final destinations (MongoDB, Elasticsearch).
• impactu/: Core utilities and shared logic for the project.
• deploys/: Deployment logic for external services (APIs, databases) via DAGs.
• backups/: Database backup automation via DAGs.
• tests/: Integration, unit, and data quality tests.

📋 Requirements and Architecture

For details on design principles (Checkpoints, Idempotency, Parallelism), see the System Requirements document.

🛠 DAG Naming Standard

To maintain consistency in the Airflow interface, we follow this convention:

⚙️ Development and Deployment

This repository focuses exclusively on DAG logic and ETL processes. The base infrastructure is provided by the Chia repository.

For details on the CI/CD strategy, image building, and environment management, see the document: 👉 README_DEVOPS.md

Local Workflow

1. Clone the repository.
2. Install dependencies: pip install -r requirements.txt.
3. Develop DAGs in the dags/ folder.
4. Validate integrity: pytest tests/etl/test_dag_integrity.py.

Colav - ImpactU