AntaresFabricFramework

The AntaresFabricFramework is an accelerator for Microsoft Fabric data ingestion, validation and transformation. It uses a configuration file to describe each source and destination and includes details for data validation and preprocessing.

Contents

• Installation
• Usage
• Features
• Contributing
• License
• Contact Information

Installation

The following options are available to install the library into a Fabric environment:

1. PyPi (Recommended): To use the latest base version available in the publish repository (PyPi), open fabric and edit or create an environment. Open the environment, click public libraries, click Add from PyPi, in the library text box type AntaresFabricFrameworkBase, then click publish.

2. Custom Library: The dist folder contains the wheel (.whl) file that can be uploaded into the Fabric environment as a custom library. To do that, open fabric and edit or create an environment. Open the environment, click custom libraries, then click Upload and upload the .whl file. Click Publish. If changes are required to the base code, follow these steps to deploy the custom library:

• Fork the repository
• Create a new branch
• Edit the code
• Build the wheel file (eg. py -m build)
• Upload as per above

To install the base notebooks required for the framework (Constants_py, PreProcessingUtils_py and OrchestrateIngestion), follow these steps:

1. Clone this repo to the destination (eg. customers devops)
2. Create a new branch
3. Create the control workspace in Fabric
4. Connect the new branch in the repo to the workspace and in the git folder section type 'FabricArtifacts' to point to that folder in the repo (this is where the notebooks and lakehouse are stored).

After it syncs, the control workspace should contain a control lakehouse and the 3 notebooks as listed earlier.

Usage

In order to run the framework, there are a few key dependencies that need to be in place:

1. Ingestion Configuration file (or table): The ingestion configuration is a json file that contains information about the source of the data, how we want to save the data, and some optional parameters we can use during the ingestion process. There is a sample file in the repository ingestion_configuration.json containing all of the attributes used during the data ingestion. After the initial execution of the framework, the file will be saved into a fabric table and read from the table each time after that. The table can be edited via SQL commands in a notebook or if the table is deleted, the json file will become the initial source again. The ingestgion configuration file and table are both located in the Control Lakehouse.

2. Keyvault: is used for storing the connection string of every source except Fabric tables and files (FABRIC-TEXT and FABRIC-TABLE read straight from OneLake and need no Key Vault entry). Secrets are read with the identity that is running the notebook: the signed-in user for a manual run, or the executing identity of the data pipeline that triggered the notebook. No tenant id, client id or client secret is stored anywhere. Grant that identity the Key Vault Secrets User role on the vault and set only key_vault_name and secret_name in the ingestion configuration.

The SQL source types and the JDBC URL each secret must hold (one secret per source, the value is the URL and nothing else):

Notes:

• For the -MFA types the URL must not contain user, password or authentication; the framework requests the token itself and rejects a secret that carries credentials. Fabric SQL endpoints do not support SQL authentication at all.
• The notebook identity needs at least Read on the Fabric item (or the equivalent database permission on Azure SQL) in addition to the Key Vault role.
• source_name for the SQL types is the full database.schema.table reference.
• Underscores and hyphens in source_type are interchangeable (AZURE_SQL_BASIC is the same as AZURE-SQL-BASIC).
• A password containing ; must be wrapped in braces in the URL, for example password={p;wd}.

3. Constants notebook: A Constants_py notebook is required to sit in the Control lakehouse and contains global constants to be used during the ingestion. These global constants are generally set once and used throughout the framework. Global constants include the names of each stage in the medallion architecture and the Azure Log Monitor details. There is a sample Constants_py notebook in the repository.

4. Azure Log Monitor (OPTIONAL): If the Azure Monitor Log service is setup, the framework can push logs from anywhere in the preprocessing stage and at predefined stages in the ingestion process. The logging is turned on via the Constants notebook, by setting the LOGGING_LEVEL to either INFO or DEBUG. NONE will turn off the logging. INFO will provide high level processing information at each major step in the ingestion. DEBUG will provide more detailed logs at higher intervals. The schema of the log table that is required to be setup is can also be found in the repository under AuditLogEntity.json

5. PreProcessing notebook (OPTIONAL): Prior to the ingestion into the first (RAW) stage, a preprocessing function can be run to modify the data coming from the source. Typically, this is used to clean up or validate manually input .csv files or to merge multiple files into a single table. This is optional and does not need to exist for the ingestion to work. To create a preprocessing function use the following function definition in the PreProcessing_py notebook:

   def preProcess(source):

   source - will be the ingestion configuration source record for a single sourceID.

6. Orchestration Notebook: The main notebook that is used to execute the ingestion process code. This note book can also contain a parameters cell that can be used to capture additiona parameters (in the form of global constants). For example, if the notebook is executed via a data pipeline in Fabric, the Data Pipeline Run ID can be passed in as a parameter and tracked via the logging module so users can trace back any errors to a specific pipeline.

To run the ingestion framework, attach the control warehouse to the orchestration notebook (Explorer > Data items > Add > Warehouse), import the library:

from FabricFramework.FabricFramework import *

then instantiate it. The only argument is the optional pre-processing function; everything else is read from the attached warehouse:

ff = FabricFramework(preProcess)   # or FabricFramework() when no pre-processing is needed

On start-up the framework reads every constant from dbo.constants_table and the source metadata from dbo.ingestion_configuration (or the table named by the TABLE_NAME constant) in the attached warehouse. Both tables are created and populated through the Antares Fabric Framework UI. If no warehouse is attached, the constants table is missing or empty, a required constant is not set, or the ingestion configuration table is missing, start-up stops with a FrameworkSetupError that says exactly what to configure. Azure Monitor logging details are optional: when any of them is missing, logging is switched off for the run and a notice is printed.

The framework ingestion process can be executed as follows:

ff.runIngestion() - this will run through the entire ingestion configuration (Every source)

You can also provide 3 parameters for more granual control over that is run as part of the ingestion:

ff.runIngestion(layer=[LAKEHOUSE_RAW_STAGE_NAME, LAKEHOUSE_TRUSTED_STAGE_NAME], system_code=['Test'], sourceID=[1, 3, 5])

layer - determines the stage that should be run (eg. RAW and TRUSTED, or just RAW or just TRUSTED). The names of each stage is determines by the constants (eg. LAKEHOUSE_RAW_STAGE_NAME) in the constants notebook

system_code - determines which specific system code(s) should be run as part of this execution (this can be a list of system codes)

sourceID - determines which sourceID's should be run on this run (if the ingestion is turned off in the ingestion configuration, the systemID will still be ignored)

Features

Contributing

Guidelines for contributing to the project:

1. Fork the repository
2. Create a new branch (git checkout -b feature-branch)
3. Commit your changes (git commit -m 'Add some feature')
4. Push to the branch (git push origin feature-branch)
5. Open a pull request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contact Information

For any questions, please contact martonm@antaressolutions.com.au.

MIT License

Copyright (c) [2024] [Antares Solutions]

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.