impactu-airflow 0.1.0

Apache Airflow DAGs and ETL processes for ImpactU

ImpactU Airflow ETL

Central repository for Apache Airflow DAGs and ETL (Extraction, Transformation, and Loading) processes for the ImpactU project. This package includes the full logic for data orchestration, source extraction, and data processing.

🚀 Description

This project orchestrates data collection from various scientific and academic sources, its processing using the Kahi tool, and its subsequent loading into query systems such as MongoDB and Elasticsearch.

� Installation

You can install the package directly from PyPI:

pip install impactu_airflow

Or for development:

git clone https://github.com/colav/impactu_airflow.git
cd impactu_airflow
pip install -e .

�📂 Project Structure

The repository is organized by data lifecycle stages and Airflow components:

• dags/: Apache Airflow DAG definitions.
• extract/: Extraction logic for sources like OpenAlex, ORCID, ROR, etc.
• transform/: Transformation and normalization processes (Kahi).
• load/: Loading scripts to final destinations (MongoDB, Elasticsearch).
• impactu/: Core utilities and shared logic for the project.
• deploys/: Deployment logic for external services (APIs, databases) via DAGs.
• backups/: Database backup automation via DAGs.
• tests/: Integration, unit, and data quality tests.

📋 Requirements and Architecture

For details on design principles (Checkpoints, Idempotency, Parallelism), see the System Requirements document.

🛠 DAG Naming Standard

To maintain consistency in the Airflow interface, we follow this convention:

Type	Format	Example
Extraction	extract_{source}	extract_openalex
Transformation	transform_{entity}	transform_sources
Loading	load_{db}_{env}	load_mongodb_production
Deployment	deploy_{service}_{env}	deploy_mongodb_production
Backup	backup_{db}_{name}	backup_mongodb_kahi
Tests	tests_{service}	tests_kahi

⚙️ Development and Deployment

This repository focuses exclusively on DAG logic and ETL processes. The base infrastructure is provided by the Chia repository.

For details on the CI/CD strategy, image building, and environment management, see the document: 👉 README_DEVOPS.md

Local Workflow

1. Clone the repository.
2. Install dependencies: pip install -r requirements.txt.
3. Develop DAGs in the dags/ folder.
4. Validate integrity: pytest tests/etl/test_dag_integrity.py.

---

Colav - ImpactU