metamorphic-relations 0.1.2

Metamorphic relations library

Installation

pip install metamorphic_relations

Get Started - MNIST Example

For more information on all classes and functions see https://3200-metamorphic-relations-lib.readthedocs.io/

Create Data Object

Stores the data that the metamorphic relations (MRs) are used on

from keras.datasets import mnist

from metamorphic_relations.Data import Data



MNIST = mnist.load_data()

data = Data(train_x=MNIST[0][0], train_y=MNIST[0][1], 

        test_x=MNIST[1][0], test_y=MNIST[1][1], max_y=10)

Create Keras Model

A deep learning keras model to be trained for MNIST image classification

from keras import Sequential

from keras.layers import Dense

import keras.layers as layers



MNIST_model = Sequential()

MNIST_model.add(Dense(128, input_shape=MNIST[0][0][0].flatten().shape))

MNIST_model.add(layers.LeakyReLU())

MNIST_model.add(Dense(256))

MNIST_model.add(layers.LeakyReLU())

MNIST_model.add(Dense(256))

MNIST_model.add(layers.LeakyReLU())

MNIST_model.add(Dense(data.max_y))

MNIST_model.compile(optimizer="adam", loss="mse", metrics=['accuracy'])

Create Metamorphic Relations (MRs)

A list of domain specific MRs (DSMRs) is created for the given dataset using the transforms provided by ImageMR

from metamorphic_relations.MR import MR

from metamorphic_relations.ImageMR import ImageMR



DSMRs = []

DSMRs += MR.for_all_labels(lambda x: ImageMR.rotate_transform(x, angle=180), [0, 1, 6, 8, 9], [0, 1, 9, 8, 6], "Rotate 180 degrees")

DSMRs += MR.for_all_labels(lambda x: ImageMR.flip_vertical_transform(x), [0, 1, 3, 8], name="Flip vertical")

DSMRs += MR.for_all_labels(lambda x: ImageMR.flip_horizontal_transform(x), [0, 1, 8], name="Flip horizontal")

The list of Generic MRs (GMRs) for images is provided by ImageMR

GMRs = ImageMR.get_image_GMRs()    

Create MRModel

The object from which the MRs can be applied to the data and the model can be trained

from metamorphic_relations.MRModel import MRModel



MR_model = MRModel(data=data, model=MNIST_model, 

    transform_x=lambda x: x.reshape((x.shape[0], -1)), 

    GMRs=GMRs, DSMRs=DSMRs)

Use MRModel to Create Additional Data

The list of MRs are converted to MR objects which can be used to create new data

MR_model.DSMRs.perform_MRs_tree(data.train_x, data.train_y, data.max_y)

The maximum number of composite MRs can be increased before perform_MRs_tree() to create more data

MR_model.DSMRs.update_composite(max_composite=2)

Use MRModel to Train Keras Model

Trains the given model and returns F1 scores and models for various configurations

from metamorphic_relations.Results import Results

MR_model.compare_MRs() compares each MR individually using all the training data

results, models = MR_model.compare_MRs()

MR_model.compare_MR_sets() compares each set of MR (No MRs, GMRs, DSMRs, GMRs + DSMRs) using all the training data

results, models = MR_model.compare_MR_sets()

MR_model.compare_MR_sets() compares each set of MR (No MRs, GMRs, DSMRs, GMRs + DSMRs) using increasing proportions of the training data

results, models = MR_model.compare_MR_sets_counts()

Save and Load Results

Result objects can be written to and read from text files

results, _ = MR_model.compare_MR_sets_counts()

results.write_to_file("Output/MNIST_sets_results.txt")

Results.read_from_file("Output/MNIST_sets_results.txt")

Present Results

Using compare_MRs() gives individual results for each MR, these can be tabulated with initial number of training instances, actual number used, train and test F1 score

results, _ = MR_model.compare_MRs()

results.write_to_file("Output/MNIST_individual_best_results.txt")

Results.read_from_file("Output/MNIST_individual_best_results.txt").print_individual()

The results for each set from compare_MR_sets_counts() can be graphed with graph_all()

results, _ = MR_model.compare_MR_sets_counts()

results.write_to_file("Output/MNIST_sets_results.txt")

Results.read_from_file("Output/MNIST_sets_results.txt").graph_all("MNIST")