Basic implementation with GPy of an Automatic Bayesian Covariance Discovery (ABCD) system
Project description
Basic implementation with GPy of an Automatic Bayesian Covariance Discovery (ABCD) system
(as in Lloyd, James Robert; Duvenaud, David Kristjanson; Grosse, Roger Baker; Tenenbaum, Joshua B.; Ghahramani, Zoubin (2014): Automatic construction and natural-language description of nonparametric regression models. In: National Conference on Artificial Intelligence, 7/27/2014, pp. 1242-1250. Available online at https://academic.microsoft.com/paper/1950803081.)
Installation
pip install GPy_ABCD
Usage
The main function exported by this package is explore_model_space; note that if the parallel argument is True then the function should be called from within a if __name__ == '__main__':
import numpy as np from GPy_ABCD import * if __name__ == '__main__': # Example data X = np.linspace(-10, 10, 101)[:, None] Y = np.cos( (X - 5) / 2 )**2 * X * 2 + np.random.randn(101, 1) # Main function call with suggested arguments best_mods, all_mods, all_exprs, expanded, not_expanded = explore_model_space(X, Y, start_kernels = standard_start_kernels, p_rules = production_rules_all, utility_function = 'BIC', restarts = 3, rounds = 2, buffer = 3, dynamic_buffer = True, verbose = False, parallel = True) # Typical output exploration printout for mod_depth in all_mods: print(', '.join([str(mod.kernel_expression) for mod in mod_depth]) + f'\n{len(mod_depth)}') print() # Explore the best 3 models in detail from matplotlib import pyplot as plt for bm in best_mods[:3]: print(bm.kernel_expression) print(bm.model.kern) print(bm.model.log_likelihood()) print(bm.cached_utility_function) bm.model.plot() print(bm.interpret()) # Perform some predictions predict_X = np.linspace(10, 15, 50)[:, None] preds = best_mods[0].predict(predict_X) print(preds) plt.show()
Importable elements from this package (refer to the section below for context):
The GPModel class
The main function explore_model_space
The model_search_rounds function to continue a search from where another left-off
Single and list model fitting functions fit_one_model, fit_model_list_not_parallel and fit_model_list_parallel
The default start kernels standard_start_kernels and production rules production_rules_all, along with the same production rules grouped by type in a dictionary production_rules_by_type
The concrete KernelExpression subclasses SumKE, ProductKE and ChangeKE
The frozensets of base_kerns and base_sigmoids
(The purpose of exporting elements in the last 3 lines is for users to create alternative sets of production rules and starting kernel lists by mixing kernel expressions and raw strings of base kernels)
Project Structure
Read the paper mentioned above for a full picture of what an ABCD system is, but, briefly, it consists in exploring a space of compositional kernels built from a few carefully selected base ones, returning the best fitting models using them and generating simple text interpretations of the fits based on the functional shapes of the final composed covariance kernels and parameter values.
The key pillars of this project’s ABCD system implementation structure are the following:
Kernels.baseKernels contains the “mathematical” base kernels (i.e. GPy kernel objects) for the whole machinery
This script also acts as a general configuration of what the system can use (including a few pre-packaged flags for certain behaviours)
Some of the base kernels are simply wrapped GPy-provided kernels (White-Noise, Constant and Squared-Exponential)
The others are either not present in GPy’s default arsenal or are improved versions of ones which are (Linear which can identify polynomial roots and purely-Periodic standard-periodic kernel)
It contains sigmoidal kernels (both base sigmoids and indicator-like ones, i.e. sigmoidal hat/well) which are not used directly in the symbolic expressions but are substituted in by change-type kernels
It contains (multiple implementations of) change-point and change-window kernels which use the aforementioned sigmoidals
KernelExpansion.kernelExpression contains the “symbolic” kernel classes constituting the nodes with which to build complex kernel expressions in the form of trees
The non-abstract kernel expression classes are SumKE, ProductKE and ChangeKE
SumKE and ProductKE are direct subclasses of the abstract class SumOrProductKE and only really differ in how they self-simplify and distribute over the other
ChangeKE could be split into separate change-point and change-window classes, but a single argument difference allows full method overlap
SumOrProductKE and ChangeKE are direct subclasses of the abstract base class KernelExpression
The above kernel expression classes have a wide variety of methods providing the following general functionality in order to make the rest of the project light of ad-hoc functions:
They self-simplify when modified through the appropriate methods (they are symbolic expressions after all)
They can produce GPy kernel objects
They can line-up with and absorb fit model parameters from a matching GPy object
They can rearrange to a sum-of-products form
They can generate text interpretations of their sum-of-products form
KernelExpansion.grammar contains the various production rules and default kernel lists used in model space exploration
Models.modelSearch contains the system front-end elements:
The GPModel class, which is where the GPy kernels/models interact with the symbolic kernel expressions
Functions to fit lists of models (the parallel version uses multiprocessing’s Pool, but alternative parallel frameworks’ versions can be implemented here)
The explore_model_space function, which is the point of it all
The model_search_rounds function, which is used by the above but also meant to continue searching by building on past exploration results
Further Notes
The important tests are in pytest scripts, but many other scripts are present and intended as functionality showcases or “tests by inspection”
Additionally, pytest.ini has a two opposite configuration lines intended to be toggled to perform “real” tests vs other “by inspection” tests
Please feel free to fork and expand this project since it is not the focus of my research and merely a component I need for part of it, therefore I will not be expanding its functionality in the near future
Possible expansion directions:
Many “TODO” comments are present throughout the codebase
Optimising ChangeWindow window-location fitting is an open issue (multiple implementations of change-window and the sigmoidal kernels they rely on have already been tried; see the commented-out declarations in baseKernels.py)
The periodic kernel could be more stable in non-periodic-data fits (GPy’s own as well)
Making each project layer accept multidimensional data, starting from the GPy kernels (some already do)
Expanding on the GPy side of things: add more methods to the kernels in order to make use of the full spectrum of GPy features (MCMC etc)
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.