Skip to main content

This library contains reusable code for various projects

Project description

Common Python Helper Functions

This library contains reused code for all the EASIER-AI projects written in Python.

Using the library

Install

This library is available through PIP package manager. To install it, execute

pip install easierai-common-functions

Importing

The library needs to be imported in order to use it:

import common_functions.helpers as helpers

from minio import Minio

from common_functions.logger import Logger

Then, there needs to be some configuration:

helpers.config = helpers.read_config_file(config_file_path)

helpers.minioClient = Minio(minio_host + ':' + minio_port, minio_access, minio_secret, secure=False)

helpers._logger = Logger('helpers', 'helpers.py')

If you wish to check the validity of the configuration provided for an inferencer, you can use this method (it will notify at start if there are no valid models):

helpers.check_initial_config(eslib, False)

Where eslib is a valid started instance of the elasticsearch library.

Necessary environmental variables

This library reads from the following environmental variables:

  • LOGSTASH_HOST: IP/hostname hosting the Logstash service to upload the logs
  • LOGSTASH_PORT: port of where the Logstash service is listening

Usage

The library has these functions available:

get_data_shape(data_type, num_features, num_samples, algorithm)

Outputs the data_shape required according to the parameters passed.

importer(algorithm, inference_type=constants.ESTIMATOR, lr=0.001)

Returns a predictor instance according to the parameters passed.

load_model_file(eslib, id, inference_type=constants.ESTIMATOR)

Returns the document stored on Elasticsearch, which includes, among other things, the h5 and pkl files where we had saved the model and the scalers, respectively. :param id: id of the entity :param inference_type: this param helps us to download the right model according to its features and parameters :return: dict with the format {"extension": object, "extension2": object2}

read_config_file(config_path)

Initializes the config variable

check_initial_config(eslib, is_classifier)

Checks if there is any model in the database that matches the configuration provided.

scale_dataset(scaler, data, i, ft_range=(-1, 1), training=True)

Scale data (in np.array or list format) using MinMaxScaler and the ft_range given (default is (-1,1) :param data: array of data to be scaled :param i: feature corresponding to the scaler :param ft_range: tuple containing minimum and maximum values of the data already scaled :return: tuple of scaler and data scaled

compose_model_params(is_classifier)

Composes a json object with the parameters of a trained model to store in the database

compose_model_params_filter(is_classifier)

Composes a json object with the parameters in the config file used to look for the models in the database.

save_model(eslib, id, metadata, dict={}, inference_type=constants.ESTIMATOR, _id=None, save_tflite=False, calibration_data=None)

Saves the model related files after training. Has the ability to save a model as tflite format. The parameter dict should come in the format {"extension1": object1, "extension2": object2 ... }. The calibration_data is only used when saving a model as tflite format, and should be a representation of the dataset.

Additional features

Constants file

The constants used on EASIER-AI services are stored in common_functions/constants.py file. It can be imported as:

import common_functions.constants as constants

Advanced logger

This logger has the same syntax as the default logging python library. It needs to be imported and initialized as:

from common_functions.logger import Logger

logger = Logger(service_name, filename)

It then can be used as logger.info(message), logger.debug(message, additional_info), etc.

This logger, apart from printing to console, uploads each log instance to Elasticsearch via Logstash, through a TCP port. To use this functionality it is needed to define the previously mentioned LOGSTASH_HOST and LOGSTASH_PORT environment variables.

Edge toolkit

This class is in charge of converting a tensorflow or keras model into tensorflow lite. It can be used as:

from edge_tools import Edge_Toolkit

edge_toolkit = Edge_Toolkit(logger)

edge_toolkit.convert_model_lite(calibration_data=calibration_data, keras_model_path=filename + '.' + constants.MODEL_EXTENSION)

After executing these lines, the tflite file will be stored in ../storage/ and can be uploaded to a remote filesystem.

Model definitions

The model definitions used by EASIER are also stored in this library. They are imported by the helpers file using the importer( ... ) function.

Copyright

Copyright (C) 2020 ATOS Spain All Rights Reserved.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for easierai-common-functions, version 1.9.2
Filename, size File type Python version Upload date Hashes
Filename, size easierai_common_functions-1.9.2-py3-none-any.whl (39.8 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size easierai_common_functions-1.9.2.tar.gz (12.7 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page