gdmo 0.0.9

GDMO native classes for standardized interaction with data objects within Azure Databricks

gdmo

GDMO native classes for standardized interaction with data objects within Azure Databricks

This custom library allows our engineering team to use standardized packages that strip away a load of administrative and repetitive tasks from their daily object interactions. The current classes supported (V0.1.0) are:

Installation

Install this library using pip:

pip install gdmo

Usage

Forecast - Forecast

Standardized way of forecasting a dataset. Input a dataframe with a Series, a Time, and a Value column, and see the function automatically select the right forecasting model and generate an output.

Example usage:

forecaster = Forecast(spark, 'Invoiced Revenue').set_columns('InvoiceDate', 'ProductCategory', 'RevenueUSD')\
                                               .set_forecast_length(forecast_length)\
                                               .set_last_data_point(lastdatamonth)\
                                               .set_input(df)\
                                               .set_growth_cap(0.02)\
                                               .set_use_cap_growth(True)\
                                               .set_modelselection_breakpoints(12, 24)\
                                               .set_track_outcome(False)\
                                               .build_forecast()

Future expansions

API - APIRequest

Class to perform a standard API Request using the request library, which allows a user to just add their endpoint / authentication / method data, and get the data returned without the need of writing error handling or need to understand how to properly build a request.

Tables - Landing

Class to land a dataframe or csv file to the databricks landing zone, and optionally convert this to the bronze layer data. Just say where to store it, and the class will take care of it with error handling associated and a normalized routine is followed.

Tables - Delta

No longer one needs to write a twelve-command notebook to create a table. Call this class once and see it happen.

Development

To contribute to this library, first checkout the code. Then create a new virtual environment:

cd gdmo
python -m venv venv
source venv/bin/activate

Now install the dependencies and test dependencies:

python -m pip install -e '.[test]'

To run the tests:

python -m pytest