AntaresFabricFramework

The AntaresFabricFramework is an accelerator for Microsoft Fabric data ingestion, validation and transformation. Every source is described by a record in a control warehouse table, and the framework moves it from the source system into RAW and TRUSTED lakehouse tables with optional validation, pseudonymisation and pre-processing.

Contents

• Installation
• Prerequisites
• Usage
• Source types and Key Vault secrets
• Optional constants
• Development
• License
• Contact Information

Installation

1. PyPI (recommended): open Fabric, edit or create an environment, select Public libraries, choose Add from PyPI, enter AntaresFabricFrameworkBase and publish.

2. Custom library: build the wheel from this repository and upload it to the environment's Custom libraries:

   py -m build
   

   The wheel is written to dist/. Upload it, then publish the environment.

Prerequisites

The framework reads everything it needs from the data warehouse attached to the orchestration notebook. Both control tables are created and populated through the Antares Fabric Framework UI.

1. dbo.constants_table: the global constants for the deployment, such as the medallion stage names and the optional Azure Monitor logging details. Stage names default to CONTROL, RAW, TRUSTED and CURATED when not set.

2. dbo.ingestion_configuration: one record per source describing where the data comes from, how it is authenticated, and where and how it is saved in RAW and TRUSTED. At minimum this table must exist. The table name can be overridden with the TABLE_NAME constant.

3. Azure Key Vault: holds the connection string for every source except Fabric tables and files. Secrets are read with the identity that runs the notebook: the signed-in user for a manual run, or the pipeline's executing identity (a user, a service principal, or the workspace identity when the Notebook activity is configured with it). Grant that identity the Key Vault Secrets User role on the vault and set only key_vault_name and secret_name on the source record. No tenant id, client id or client secret is stored anywhere.

4. Azure Monitor logging (optional): set LOGGING_LEVEL to INFO or DEBUG together with LOGS_AZURE_TENANT_ID, LOGS_AZURE_CLIENT_ID, LOGS_AZURE_CLIENT_SECRET, DCE_ENDPOINT, DRC_ID and LOG_STREAM. If any of them is missing, logging is switched off for the run and a notice is printed. The log stream receives these columns: LogRunID, UserName, Group, SourceID, Entity, Layer, Source, Target, StartDateTime, EndDateTime, LogDateTime, LogType, Status, SourceRows, TargetRows, Error, ValidationRules, Parameters.

5. Pre-processing notebook (optional): a function run against the source before it is saved to RAW, typically to clean manually maintained files or merge several files into one table. Define it as:

   def preProcess(source):
       ...
   

   where source is the ingestion configuration record for the SourceID being processed.

Usage

Attach the control warehouse to the orchestration notebook (Explorer > Data items > Add > Warehouse), then:

from FabricFramework.FabricFramework import *

ff = FabricFramework(preProcess)   # or FabricFramework() when no pre-processing is needed
ff.runIngestion()

The only constructor argument is the optional pre-processing function. On start-up the framework locates the attached warehouse, reads the constants and the ingestion configuration, and stops with a FrameworkSetupError that says exactly what to configure if no warehouse is attached, the constants table is missing or empty, a required constant is not set, or the ingestion configuration table is missing.

runIngestion() with no arguments processes every enabled source from the source system through to TRUSTED. Three optional parameters narrow the run:

ff.runIngestion(layer=['RAW', 'TRUSTED'], system_code=['Sales'], sourceID=[1, 3, 5])

• layer: which stages to run, using the stage names from the constants table.
• system_code: only sources with these system codes.
• sourceID: only these SourceIDs. A source disabled in the configuration is still skipped.

While a run is in progress a per-source progress dashboard is rendered in the notebook, and a plain-text summary of any failed sources is printed when the run finishes.

Source types and Key Vault secrets

Underscores and hyphens in source_type are interchangeable, so AZURE_SQL_BASIC and AZURE-SQL-BASIC are the same type. The old catch-all SQL type is rejected with a message naming its replacements.

All eight go through the same RAW and TRUSTED pipeline once loaded: optional pre-processing, the all-NULL row filter, the configured raw_savetype, then validation or a direct copy into TRUSTED with the configured trusted_savetype.

Key Vault secret formats

Each secret is a single string. For the SQL types it is the JDBC URL and nothing else.

Notes:

• For the -MFA types the URL must not contain user, password or authentication; the framework requests the token itself and rejects a secret that carries credentials. Fabric SQL endpoints do not support SQL authentication at all.
• The notebook identity needs at least Read on the Fabric item, or the equivalent database permission on Azure SQL, in addition to the Key Vault role.
• source_name for the SQL types is the full database.schema.table reference.
• A password containing ; must be wrapped in braces in the URL, for example password={p;wd}.
• List-valued columns such as primary_key, data_validation and preprocessing_options may be left NULL; NULL is treated as an empty list.

Optional constants

Development

The unit tests run locally without a Fabric connection. Create a virtual environment with Python 3.11, install pytest and pyspark==3.5.3, and run:

python -m pytest tests/validation -q

On Windows set PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON to the environment's interpreter first so Spark workers use the same Python.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contact Information

For any questions, please contact martonm@antaressolutions.com.au.

MIT License

Copyright (c) [2024] [Antares Solutions]

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.