Skip to main content

al-data-core

Generic Airflow data pipeline engine. Install once, configure per project.

Python and Setup

  • Python: 3.10+
  • Packaging: setuptools (src layout)

Install locally in editable mode:

pip install -e .

Requires al-data-utils to be installed first.

Project Structure

  • src/al_data_core

Main modules under src/al_data_core:

  • config_processor — DataCoreConfig
  • tasks_processor — TaskProcessor
  • azure_as_processor — run_azure_as_refresh
  • external_handler_base — ExternalHandlerBase, discover_handlers (used for XLS and API handlers)
  • dags_installer — CLI for copying DAG templates

DAG Templates

After installing, copy the ready-to-use DAG templates to your Airflow dags folder:

al-data-core copy-dags                          # latest → /opt/airflow/dags
al-data-core copy-dags /my/dags                 # custom path
al-data-core copy-dags --version v2.0.0         # specific Airflow version
al-data-core list-versions                      # show available versions

Available versions:

  • v2.0.0 — Airflow 2.x
  • v3.0.0 — Airflow 3.x

Two files are copied:

  • data_core_dag.py — main DAG file, reads data_core_tasks_config.yaml and registers all project DAGs
  • data_core_config_engine_dag.py — runs every 15 min, queries config DB and YAML, writes data_core_tasks_config.yaml

Project Layout (per-project repo)

dags/
├── data_core_dag.py                    <- copied from template
├── data_core_config_engine_dag.py      <- copied from template
└── data_core/
    ├── data_core_config.yaml           <- project config
    ├── .configs/data_core/data_core_tasks_config.yaml  <- generated by config engine dag
    ├── data_core_handler_sql/
    │   └── sql_queries/
    │       └── <scope>/
    │           └── <param_code>.sql
    ├── data_core_handler_xls/          <- optional, project-specific
    └── data_core_handler_api/          <- optional, project-specific

data_core_config.yaml

CONFIG:
  config_connection_variable: my_mssql_secret   # Airflow variable name
  mappings:
    config_get_tables_list:          integrations_db.config.get_tables_list
    config_parameters:               integrations_db.config.parameters
    config_fn_get_param_values:      integrations_db.config.fn_get_param_values
    config_calc_date_ranges:         integrations_db.config.calc_date_ranges
    config_update_param_values:      integrations_db.config.update_param_values

DAGS:
  my_project_ods_dag:
    schedule_interval: '5 */1 * * *'
    scopes: ['ODS_SOURCE']
    tags: ['my_project']
  my_project_dw_dag:
    schedule_interval: '30 */1 * * *'
    scopes: ['DW_TARGET']
    tags: ['my_project']
    azure_as:                         # optional: refresh semantic model after ETL
      connection_variable: my_azure_as_secret
      models:
        - name: MyModel
          refresh_request: {Type: Full, CommitMode: transactional, MaxParallelism: 2, RetryCount: 2, Objects: []}

ODS_SOURCE:
  enabled: true
  source_type: mssql
  source_connection_variable: source_mssql_secret
  destination_connection_variable: dest_mssql_secret

DW_TARGET:
  enabled: true
  destination_connection_variable: dest_mssql_secret

XLS:
  enabled: true
  handler: data_core_handler_xls   # opts this scope into handler-based dispatch instead of seq_code-based copy/merge

ods_britix_api:
  enabled: true
  handler: data_core_handler_api/britix24

Handlers

A scope opts into handler-based dispatch by setting handler to its handlers folder (relative to data_core_config.yaml, or an absolute path) — the value is the path, so TaskProcessor doesn't hardcode scope_code names and adding a new handler scope needs only config, never a template change. Two layouts are supported:

  • One file per param_code — file name matches the task's param_code exactly.
  • One generic handler per scope — when the folder holds a single handler file, it serves every param_code in the scope and distinguishes tasks via self.context.param_code.
# dags/data_core/data_core_handler_xls/fact_sales_plan.py
from al_data_core.external_handler_base import ExternalHandlerBase

class MySalesHandler(ExternalHandlerBase):
    def proceed(self) -> None:
        ...
# dags/data_core/data_core_handler_api/britix24/bitrix_ingest.py
from al_data_core.external_handler_base import ExternalHandlerBase

class BitrixIngestHandler(ExternalHandlerBase):
    def proceed(self) -> None:
        scope = self.context.scope_config
        entity = scope['entities'][self.context.param_code]
        ...

TaskProcessor constructs the handler with no arguments, sets handler.context to a HandlerContext (scope_code, param_code, scope_config, config_file_path, params_details, debug), then calls proceed() — handlers read run-time settings from self.context instead of hardcoding them. Discovery happens at task-run time, not DAG-parse time, so a broken handler file can't fail the whole DAG file's parse and handler modules aren't re-imported every scheduler cycle. Each file must define exactly one ExternalHandlerBase subclass.

Development Guides

  • RELEASING.md — release flow and Azure Blob publish steps

Changelog

See CHANGELOG.md for release notes.

Metadata

Release files for al-data-core 0.6.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for al-data-core 0.6.2
File Size Uploaded Provenance Actions
al_data_core-0.6.2.tar.gz 18.9 kB Without provenance Details for al_data_core-0.6.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for al-data-core 0.6.2
File Interpreter ABI Platform Actions
al_data_core-0.6.2-py3-none-any.whl Python 3 none any Details for al_data_core-0.6.2-py3-none-any.whl

Total release size: 42.0 kB

Release files / al_data_core-0.6.2.tar.gz

Download URL al_data_core-0.6.2.tar.gz
Size 18.9 kB
Tags Source
SHA-256 checksum
How to use checksums
c02ad71818f821f38ea73ab55eb0b76929806471a88a8730a7652c8ee1320798
BLAKE2b-256 checksum
How to use checksums
416638ef86bb5ad3e677e10a5da95ace543adf8a4088308bd091fc116c787dcb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Release files / al_data_core-0.6.2-py3-none-any.whl

Download URL al_data_core-0.6.2-py3-none-any.whl
Size 23.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
c9925d6e2d166b9c4c7194ab6abc68382e13a9efc4f64ac3d6868f6ec4bff8aa
BLAKE2b-256 checksum
How to use checksums
eaf9a6eef8acfcf10da42e9a5c443a5f4f0b3c9dac89b872b4ae84f8834fc1c5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Security for al-data-core 0.6.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page