northgravity-utils 0.0.2

Python Utils for NorthGravity platform tasks

NORTH GRAVITY PYTHON UTILS

This document describes the North Gravity Python Utils package which enables users to use the NG platform tools and most common functions within their python scripts / tasks repositories.

The Python Utils can be used within:

• a single python script that is ran thanks to the Python Runner task within a pipeline in the NG application

• a single Jupyter Notebook that is ran thanks to the Jupyter Runner task within a pipeline in the NG application

• an ensemble of python scripts that are part of a container, for a Task created by the user, used in a pipeline in the NG application

NG_Utils cover most frequently used functions for data/files handling purposes.

The scope of the NG_Utils:

• Data Handler - splitting data based on train/test labels, handling dates in datasets

• File Handler - downloading/uploading model-specific datasets from/to the data lake, handling datasets formats

• Back Test - preparing short or extended backtest based on models results

How to install and set the package:

Install

pip3 install northgravity_utils==0.0.2

As the library is available from pip, it can be installed as a specific version within a Python Task from within requirements.txt just by adding:

northgravity_utils==0.0.2

The package relies on the requests library so, in the project, the user must install this library in the requirements.txt file.

pip3 install requests==2.27.1

Environment Variables

The package uses information from the environment variables. They are necessery for functionality of Nortygravity SDK package, used in Northgravity Utils Environment variables are automatically provided when running a script within a pipeline (as a Task or within the Python/Jupyter Runners). If running locally the script, users must set them in the project to be able to run the project locally.

Mandatory environment variables to set:

• LOGIN → login received from NG

• PASSWORD → password to log in. Credentials are used to generate the token so that each request is authenticated.

• NG_API_ENDPOINT → the URL to the NG platform API (by default, the url is set to https://api.northgravity.com)

---

Data Handler

Import NG_Utils and load DataHandler, FileHandler and BackTest

import ng_utils as ng_u

dth = ng_u.DataHandler()
dfh = ng_u.FileHandler()
dbt = ng_u.BackTest()

Train/Test/Val

import ng_utils as ng_u

dth = ng_u.DataHandler()

# Train/Validation/Test
tvt_dict = dth.train_test_split(features_df, target_df, val=True)

# Train/Test
tt_dict = dth.train_test_split(features_df, target_df, val=True)

Get date column

# Get date column
X_dataset = tvt_dict['Val'][0]
date_col = dth.get_date_col(X_dataset)

Shift the dates by the specified period

# Business Days
data_shifted_bd = dth.date_shift(date_col, period=2, freq='B')

# Week
data_shifted_w = dth.date_shift(date_col, period=1, freq='W')

# Quarter
data_shifted_q = dth_date_shift(date_col, period=1, freq='Q')

Shift the holiday dates based on uploaded calendar

# hol_list - holiday calendar list
# date_col - column with dates
data_shifted_calendar = dth.holiday_shift(date_col, hol_list)

Prepare index for forecast and test datasets

# Index for forecast and test
idx = dth.prep_prediction_date(target_df.index, period=3)

Prepare forecast date in DataPrep required format

# Date in DataPrep required format
# idx - dates
date_prep = dth.forecast_date_format(idx)

Model' score - regression

# Regression
actual = df_test_reg['Real']
predicted = df_test_reg['Predicted']
print(dth.get_scores_regression(actual, predicted))

Model's score - classification

# Classification
actual = df_test_class['Real']
predicted = df_test_class['Predicted']
print(dth.get_scores_classification(actual, predicted))

FileHandler

Download features and target datasets

# Features
features_df = dfh.features_download()

# Target
target_df = dfh.target_download()

Upload general models output

print(fh.forecast_upload(df_for, group_name, model_out_name))
print(fh.test_upload(df_test, group_name, model_out_name))
print(fh.drivers_upload(df_drv, group_name, model_out_name))

Convert NSCV file to Horizontal type file

## Convert NCSV file to Horizontal type file
print('NCSV type file: {}'.format(df_ncsv.tail(10)))
n2h = fh.ncsv2horizontal(df_ncsv)
print(n2h.tail(10))
print('NCSV converted to Horizontal type file')
n2h.to_csv(output_path + '/ncsv2horizontal_1.csv')

BackTests

Short BackTest

short_bt = bt.short_back_test(df=df_to_backtest, open_col='Price', trading_signal_col='Signal')
print('Short BackTest: {}'.format(short_bt))

Long Backtest

long_bt = bt.long_back_test(df=df_to_backtest, number_of_contracts=100, open_col='Price', trading_signal_col='Signal')
print('Long BackTest: {}'.format(long_bt))

Who do I talk to?

• Admin: NorthGravity [e-mail]