FoldLayer 0.1.0

A package for fold layers in neural networks

Fold Layer

The Fold Layer is a novel neural network layer designed to transform high-dimensional data with efficient learning and reduced complexity. Inspired by principles of geometric folding, this model employs a custom nonlinearity to reshape data through learned hyperplanes, allowing for non-linear transformations that improve prediction speed and convergence.

Table of Contents

• Overview
• Features
• Architecture
• Installation
• Usage
• Examples
• Research & Development
• Contributing
• License
• Workflow

Overview

The Fold Layer architecture introduces a data transformation process similar to folding in origami, where input data is sequentially mapped across high-dimensional hyperplanes. This process enables:

• A reduced need for large, high-parameter layers in standard architectures.
• Accelerated prediction through hyperplane-based data folding.
• Simplified optimization, using fewer trainable parameters to capture complex data patterns.

Features

• Custom Nonlinearity: Each layer learns a set of hyperplanes to reshape the data dynamically, reducing training time.
• Optimized Performance: Through reduced parameterization, fold layers achieve lower latency at run time.
• Configurable Optimizers: Support for various optimizers to best match the model's fold-based architecture.
• Improved Memory Efficiency: Fewer parameters make Fold Layers more memory and parameter efficient.
• Interpretability: Fold operations provide a more intuitive and interpretable alternative to traditional deep learning models.

Architecture

The fold layers are inspired by origami and the Fold and Cut Theorem to emulate a ReLU function but add the capability to fold data into more separable forms in fewer steps. They modify the data by finding the projection of the data onto a hyperplane and then adding the projection to the data twice in N-dimensional space to 'fold' the data. This process is repeated for each fold layer in the model. The fold layer is defined by the following equation:

$$ L_p(\mathbf{x}, \mathbf{n}p) = \mathbf{x} - 2 \left( \mathbb{1}{{\mathbf{n}_p \cdot \mathbf{x} > \mathbf{n}_p \cdot \mathbf{n}_p}} \right) \left(1 - \frac{\mathbf{x} \cdot \mathbf{n}_p}{\lVert \mathbf{n}_p \rVert} \right) \mathbf{n}_p $$

Installation

You can install this package from PyPi with

pip install FoldLayer

or

python -m pip install --upgrade FoldLayer

Usage

This section is under development. Please check back soon for usage instructions.

import fold_layer

Research and Development

The Fold Layer is under active research, focusing on:

• Experimenting with different fold depth and width configurations.
• Testing efficiency gains in prediction for natural language processing, computer vision, and other domains.
• Identifying optimizers that best support fold-layer dynamics.
• Experimenting with folds in higher dimensions.
• Developing a fold version of convolutional neural networks.

Contributing

Contributions to improve the Fold Layer, fix bugs, or add features are welcome! Please open an issue or submit a pull request.

Current Contributors:

Dallin Stewart - [e-mail]

Sam Layton

Jeddy Bennett - [e-mail]

Nathaniel Driggs

License

This project is licensed under the MIT License. See the LICENSE file for details.