al-data-core
Generic Airflow data pipeline engine. Install once, configure per project.
Install
pip install al-data-core
Python 3.10+. al-data-utils is installed automatically. airflow, pandas, sqlalchemy, pyodbc are expected from the host environment.
Modules
config_processor-DataCoreConfigtasks_processor-TaskProcessorazure_as_processor-run_azure_as_refreshexternal_handler_base-ExternalHandlerBase,discover_handlers(used for XLS and API handlers)dags_installer- CLI for copying DAG templates
DAG Templates
After installing, copy the ready-to-use DAG templates to your Airflow dags folder:
al-data-core copy-dags # latest → /opt/airflow/dags
al-data-core copy-dags /my/dags # custom path
al-data-core copy-dags --version v2.0.0 # specific Airflow version
al-data-core list-versions # show available versions
Available versions:
v2.0.0- Airflow 2.xv3.0.0- Airflow 3.x
Two files are copied:
data_core_dag.py- main DAG file, readsdata_core_tasks_config.yamland registers all project DAGsdata_core_config_engine_dag.py- runs every 15 min, queries config DB and YAML, writesdata_core_tasks_config.yaml
Project Layout (per-project repo)
dags/
├── data_core_dag.py <- copied from template
├── data_core_config_engine_dag.py <- copied from template
└── data_core/
├── data_core_config.yaml <- project config
├── .configs/data_core/data_core_tasks_config.yaml <- generated by config engine dag
├── data_core_handler_sql/
│ └── sql_queries/
│ └── <scope>/
│ └── <param_code>.sql
├── data_core_handler_xls/ <- optional, project-specific
└── data_core_handler_api/ <- optional, project-specific
data_core_config.yaml
CONFIG:
config_connection_variable: my_mssql_secret # Airflow variable name
mappings:
config_get_tables_list: integrations_db.config.get_tables_list
config_parameters: integrations_db.config.parameters
config_fn_get_param_values: integrations_db.config.fn_get_param_values
config_calc_date_ranges: integrations_db.config.calc_date_ranges
config_update_param_values: integrations_db.config.update_param_values
DAGS:
my_project_ods_dag:
schedule_interval: '5 */1 * * *'
scopes: ['ODS_SOURCE']
tags: ['my_project']
my_project_dw_dag:
schedule_interval: '30 */1 * * *'
scopes: ['DW_TARGET']
tags: ['my_project']
azure_as: # optional: refresh semantic model after ETL
connection_variable: my_azure_as_secret
models:
- name: MyModel
refresh_request: {Type: Full, CommitMode: transactional, MaxParallelism: 2, RetryCount: 2, Objects: []}
ODS_SOURCE:
enabled: true
source_type: mssql
source_connection_variable: source_mssql_secret
destination_connection_variable: dest_mssql_secret
DW_TARGET:
enabled: true
destination_connection_variable: dest_mssql_secret
XLS:
enabled: true
handler: data_core_handler_xls # opts this scope into handler-based dispatch instead of seq_code-based copy/merge
ods_britix_api:
enabled: true
handler: data_core_handler_api/britix24
Handlers
A scope opts into handler-based dispatch by setting handler to its handlers folder (relative to data_core_config.yaml, or an absolute path) - the value is the path, so TaskProcessor doesn't hardcode scope_code names and adding a new handler scope needs only config, never a template change. Two layouts are supported:
- One file per param_code - file name matches the task's
param_codeexactly. - One generic handler per scope - when the folder holds a single handler file, it serves every param_code in the scope and distinguishes tasks via
self.context.param_code.
# dags/data_core/data_core_handler_xls/fact_sales_plan.py
from al_data_core.external_handler_base import ExternalHandlerBase
class MySalesHandler(ExternalHandlerBase):
def proceed(self) -> None:
...
# dags/data_core/data_core_handler_api/britix24/bitrix_ingest.py
from al_data_core.external_handler_base import ExternalHandlerBase
class BitrixIngestHandler(ExternalHandlerBase):
def proceed(self) -> None:
scope = self.context.scope_config
entity = scope['entities'][self.context.param_code]
...
TaskProcessor constructs the handler with no arguments, sets handler.context to a HandlerContext (scope_code, param_code, scope_config, config_file_path, params_details, debug), then calls proceed() - handlers read run-time settings from self.context instead of hardcoding them. Discovery happens at task-run time, not DAG-parse time, so a broken handler file can't fail the whole DAG file's parse and handler modules aren't re-imported every scheduler cycle. Each file must define exactly one ExternalHandlerBase subclass.
Changelog
See CHANGELOG.md in the repository for release notes.
Metadata
Release files for al-data-core 0.6.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| al_data_core-0.6.5.tar.gz | 19.7 kB | Without provenance | Details for al_data_core-0.6.5.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| al_data_core-0.6.5-py3-none-any.whl | Python 3 | none | any | Details for al_data_core-0.6.5-py3-none-any.whl |
Total release size: 43.4 kB
Release files / al_data_core-0.6.5.tar.gz
| Download URL | al_data_core-0.6.5.tar.gz |
|---|---|
| Size | 19.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
590446f4992541ac7b714830ed7fbc155f241ac9f6d3ff51dc3801b511434c1e
|
|
BLAKE2b-256 checksum How to use checksums |
694db7b5facbea341efacc7f26233b57287b7425893dd452d97c5b58bf108af6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.7
|
Release files / al_data_core-0.6.5-py3-none-any.whl
| Download URL | al_data_core-0.6.5-py3-none-any.whl |
|---|---|
| Size | 23.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
df089a9150c9cd40e6bbb44be00e645cfa1c8eef002bf7b92dc18791252a531c
|
|
BLAKE2b-256 checksum How to use checksums |
7ef7011aced042e009fdb0b9098327cfb8236586eb94daae768df7ccbecfb814
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.7
|
Security for al-data-core 0.6.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.