Skip to main content

al-data-core

Generic Airflow data pipeline engine. Install once, configure per project.

Install

pip install al-data-core

Python 3.10+. al-data-utils is installed automatically. airflow, pandas, sqlalchemy, pyodbc are expected from the host environment.

Modules

  • config_processor - DataCoreConfig
  • tasks_processor - TaskProcessor
  • azure_as_processor - run_azure_as_refresh
  • external_handler_base - ExternalHandlerBase, discover_handlers (used for XLS and API handlers)
  • dags_installer - CLI for copying DAG templates

DAG Templates

After installing, copy the ready-to-use DAG templates to your Airflow dags folder:

al-data-core copy-dags                          # latest → /opt/airflow/dags
al-data-core copy-dags /my/dags                 # custom path
al-data-core copy-dags --version v2.0.0         # specific Airflow version
al-data-core list-versions                      # show available versions

Available versions:

  • v2.0.0 - Airflow 2.x
  • v3.0.0 - Airflow 3.x

Two files are copied:

  • data_core_dag.py - main DAG file, reads data_core_tasks_config.yaml and registers all project DAGs
  • data_core_config_engine_dag.py - runs every 15 min, queries config DB and YAML, writes data_core_tasks_config.yaml

Project Layout (per-project repo)

dags/
├── data_core_dag.py                    <- copied from template
├── data_core_config_engine_dag.py      <- copied from template
└── data_core/
    ├── data_core_config.yaml           <- project config
    ├── .configs/data_core/data_core_tasks_config.yaml  <- generated by config engine dag
    ├── data_core_handler_sql/
    │   └── sql_queries/
    │       └── <scope>/
    │           └── <param_code>.sql
    ├── data_core_handler_xls/          <- optional, project-specific
    └── data_core_handler_api/          <- optional, project-specific

data_core_config.yaml

CONFIG:
  config_connection_variable: my_mssql_secret   # Airflow variable name
  mappings:
    config_get_tables_list:          integrations_db.config.get_tables_list
    config_parameters:               integrations_db.config.parameters
    config_fn_get_param_values:      integrations_db.config.fn_get_param_values
    config_calc_date_ranges:         integrations_db.config.calc_date_ranges
    config_update_param_values:      integrations_db.config.update_param_values

DAGS:
  my_project_ods_dag:
    schedule_interval: '5 */1 * * *'
    scopes: ['ODS_SOURCE']
    tags: ['my_project']
  my_project_dw_dag:
    schedule_interval: '30 */1 * * *'
    scopes: ['DW_TARGET']
    tags: ['my_project']
    azure_as:                         # optional: refresh semantic model after ETL
      connection_variable: my_azure_as_secret
      models:
        - name: MyModel
          refresh_request: {Type: Full, CommitMode: transactional, MaxParallelism: 2, RetryCount: 2, Objects: []}

ODS_SOURCE:
  enabled: true
  source_type: mssql
  source_connection_variable: source_mssql_secret
  destination_connection_variable: dest_mssql_secret
  copy_method: pyodbc              # optional, default pyodbc

DW_TARGET:
  enabled: true
  destination_connection_variable: dest_mssql_secret

XLS:
  enabled: true
  handler: data_core_handler_xls   # opts this scope into handler-based dispatch instead of seq_code-based copy/merge

ods_britix_api:
  enabled: true
  handler: data_core_handler_api/britix24

Copy method

copy_method sets how an A0 copy task moves rows from source to destination. It is set per scope. Leave it out to get the default. An unknown value fails the task with ValueError.

Value Path
pyodbc (default) Streams rows from the source cursor and inserts them with pyodbc fast_executemany, with no pandas. The whole copy runs in one transaction. Values are exact for decimal(38), bigint, binary, text/ntext/image and datetimeoffset.

Known limits:

  • datetime2(7) and time(7) lose the 7th fractional digit (100 ns), because Python datetime stops at microseconds.
  • A destination decimal narrower than the source value fails loudly (Converting decimal loses precision) rather than rounding.
  • Invalid UTF-16 in a source string fails loudly (UnicodeDecodeError).

Handlers

A scope opts into handler-based dispatch by setting handler to its handlers folder (relative to data_core_config.yaml, or an absolute path) - the value is the path, so TaskProcessor doesn't hardcode scope_code names and adding a new handler scope needs only config, never a template change. Two layouts are supported:

  • One file per param_code - file name matches the task's param_code exactly.
  • One generic handler per scope - when the folder holds a single handler file, it serves every param_code in the scope and distinguishes tasks via self.context.param_code.
# dags/data_core/data_core_handler_xls/fact_sales_plan.py
from al_data_core.external_handler_base import ExternalHandlerBase

class MySalesHandler(ExternalHandlerBase):
    def proceed(self) -> None:
        ...
# dags/data_core/data_core_handler_api/britix24/bitrix_ingest.py
from al_data_core.external_handler_base import ExternalHandlerBase

class BitrixIngestHandler(ExternalHandlerBase):
    def proceed(self) -> None:
        scope = self.context.scope_config
        entity = scope['entities'][self.context.param_code]
        ...

TaskProcessor constructs the handler with no arguments, sets handler.context to a HandlerContext (scope_code, param_code, scope_config, config_file_path, params_details, debug), then calls proceed() - handlers read run-time settings from self.context instead of hardcoding them. Discovery happens at task-run time, not DAG-parse time, so a broken handler file can't fail the whole DAG file's parse and handler modules aren't re-imported every scheduler cycle. Each file must define exactly one ExternalHandlerBase subclass.

Changelog

See CHANGELOG.md in the repository for release notes.

Metadata

Release files for al-data-core 0.7.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for al-data-core 0.7.0
File Size Uploaded Provenance Actions
al_data_core-0.7.0.tar.gz 21.5 kB Without provenance Details for al_data_core-0.7.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for al-data-core 0.7.0
File Interpreter ABI Platform Actions
al_data_core-0.7.0-py3-none-any.whl Python 3 none any Details for al_data_core-0.7.0-py3-none-any.whl

Total release size: 46.8 kB

Release files / al_data_core-0.7.0.tar.gz

Download URL al_data_core-0.7.0.tar.gz
Size 21.5 kB
Tags Source
SHA-256 checksum
How to use checksums
ae4969dc0b4998b8525e0487596d48531d5f48b044d77de065ae854f03cb54e0
BLAKE2b-256 checksum
How to use checksums
24b8b6ce711a7e8a45336d8860f855843e8e5be82da124ea1790a015860bd505
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.7

Release files / al_data_core-0.7.0-py3-none-any.whl

Download URL al_data_core-0.7.0-py3-none-any.whl
Size 25.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
c2749a1ff3fc86b9398aa3ac7708ab7ae9a9b7a7325bd8bc8efe350e70d76b51
BLAKE2b-256 checksum
How to use checksums
b2f7b6591bdd0ec53d6f440d0ccae5222de69eeed770bcc17f54955c5d91fdff
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.7

Security for al-data-core 0.7.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page