oscars-toolbox 0.1.0

A package for optimization. See the GitHub repo for instructions and version notes.

oscars-toolbox

A package for helpful general algorithms I've developed. See the PyPI release: https://pypi.org/project/oscars-toolbox/. See also my wesbite, https://oscars47.github.io/.

Current functions as of latest version:

trabbit

This repository contains a custom gradient descent algorithm called "Tortoise and Rabbit" (trabbit) implemented in Python. The algorithm aims to perform double optimization to determine the best parameters for a given loss function by incorporating both gradient descent and random input generation strategies.

Functions and Their Usage

trabbit

The trabbit function implements the custom gradient descent algorithm. It optimizes the parameters of a given loss function using a combination of gradient descent and random input generation.

• Parameters:

  • loss_func: A function to minimize. Assumes all arguments are already passed through partial.
  • random_gen: A function to generate random inputs.
  • bounds: Bounds for each parameter (default: None). If None, no bounds are implemented.
  • x0_ls: Initial guesses within a list (default: None). If None, random_gen is used. Can also be a list of initial parameters to try before implementing gradient descent.
  • num: Number of iterations (default: 1000).
  • alpha: Learning rate (default: 0.3).
  • temperature: Fraction of iterations to use for random input generation (default: 0.1).
  • tol: Tolerance for convergence. The algorithm stops if the loss is less than tol (default: 1e-5).
  • grad_step: Step size to estimate the gradient (default: 1e-8).
  • verbose: Whether to print out the loss at each iteration (default: True).

• Returns:

  • x_best: Best parameters found.
  • loss_best: Best loss achieved.

• Example Usage:

  from oscars_toolbox.trabbit import trabbit
  
  # Define a sample loss function
  def sample_loss(x):
      return np.sum(x**2)
  
  # Define a random input generator
  def random_gen():
      return np.random.uniform(-10, 10, size=3)
  
  # Run the trabbit algorithm
  best_params, best_loss = trabbit(sample_loss, random_gen)
  print(f'Best parameters: {best_params}')
  print(f'Best loss: {best_loss}')
  

Detailed Description

The trabbit function incorporates a combination of gradient descent and random input generation to optimize a loss function. The algorithm proceeds as follows:

1. Initial Guess:

   • If x0_ls is provided, each initial guess is evaluated using a minimization function (min_func). If x0_ls is None, random inputs are generated using random_gen.

2. Minimization Function:

   • The min_func uses the Nelder-Mead algorithm (or bounded optimization if bounds are provided) to minimize the loss function and return the optimal parameters.

3. Gradient Descent with Random Hopping:

   • The algorithm performs gradient descent with a specified learning rate (alpha). If no improvement is seen for a specified fraction of iterations (temperature), the algorithm hops out and uses a new random input generated by random_gen.

4. Convergence Check:

   • The algorithm checks if the gradient is too small or if the loss is below the tolerance level (tol). If so, it hops out or terminates.

5. Verbose Output:

   • If verbose is True, the algorithm prints the current loss, best loss, and iteration details at each step.

6. Keyboard Interrupt Handling:

   • The algorithm gracefully handles keyboard interrupts and prints the best parameters and corresponding loss found so far.

implement_torch

The primary functions include training a model, evaluating its performance, and counting the number of trainable parameters. The functions utilize popular libraries such as torch, torch.nn, torch.optim, tqdm, and sklearn.

Functions

1. train_only

This function trains a given model on the provided training data and evaluates it on the validation data.

• Parameters:

  • model: The neural network model to be trained.
  • device: The device to use for computation (cpu or cuda).
  • train_loader: DataLoader for the training dataset.
  • val_loader: DataLoader for the validation dataset.
  • num_epochs: Number of training epochs (default: 5).
  • learning_rate: Learning rate for the optimizer (default: 1e-3).
  • weight_decay: Weight decay (L2 regularization) factor for the optimizer (default: 1e-4).
  • loss_func: Loss function to use (default: nn.CrossEntropyLoss()).

• Returns:

  • model: The trained model.
  • train_accuracy: Training accuracy.
  • val_accuracy: Validation accuracy.
  • train_acc_ls: List of training accuracies for each epoch.
  • val_acc_ls: List of validation accuracies for each epoch.

• Example Usage:

  model, train_accuracy, val_accuracy, train_acc_ls, val_acc_ls = train_only(model, device, train_loader, val_loader)
  

2. train_model

This function constructs and trains a model based on the provided architecture and training settings. This differs from train_only because it only

• Parameters:

  • model_func: Function to create the model.
  • device: The device to use for computation (cpu or cuda).
  • train_loader: DataLoader for the training dataset.
  • val_loader: DataLoader for the validation dataset.
  • input_size: Size of the input layer.
  • output_size: Size of the output layer.
  • neurons_ls: List specifying the number of neurons in each hidden layer.
  • num_epochs: Number of training epochs (default: 5).
  • learning_rate: Learning rate for the optimizer (default: 1e-3).
  • weight_decay: Weight decay (L2 regularization) factor for the optimizer (default: 1e-4).
  • use_cnn: Boolean flag to indicate if a convolutional neural network (CNN) should be used (default: False).
  • loss_func: Loss function to use (default: nn.CrossEntropyLoss()).
  • img_channels: Number of image channels (default: 3).

• Returns:

  • model: The trained model.

• Example Usage:

  model = train_model(my_model_func, device, train_loader, val_loader, input_size, output_size, neurons_ls)
  

3. evaluate

This function evaluates a trained model on the test data and returns the confusion matrix and optionally other metrics. Only use if using the models defined in torch_models.py.

• Parameters:

  • model: The trained neural network model.
  • test_loader: DataLoader for the test dataset.
  • num_classes: Number of classes in the dataset.
  • device: The device to use for computation (cpu or cuda).
  • return_extra_metrics: Boolean flag to indicate if additional metrics (accuracy, precision, recall, F1 score) should be returned (default: False).

• Returns:

  • conf_matrix: Confusion matrix.
  • accuracy: Accuracy score (if return_extra_metrics is True).
  • precision: Precision score (if return_extra_metrics is True).
  • recall: Recall score (if return_extra_metrics is True).
  • f1: F1 score (if return_extra_metrics is True).

• Example Usage:

  conf_matrix, accuracy, precision, recall, f1 = evaluate(model, test_loader, num_classes, device, return_extra_metrics=True)
  

4. count_parameters_torch

This function counts the number of trainable parameters in a model.

• Parameters:

  • model: The neural network model.

• Returns:

  • num_params: The number of trainable parameters.

• Example Usage:

  num_params = count_parameters_torch(model)
  print(f'The model has {num_params} trainable parameters.')
  

Example Workflow

from oscars_toolbox.implement_torch import train_only, evaluate
import matplotlib.pyplot as plt

# Assuming `train_loader`, `val_loader`, `test_loader` are defined DataLoader objects
# and `my_model_func` is a function that creates a neural network model

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Train the model
model, train_accuracy, val_accuracy, train_acc_ls, val_acc_ls = train_only(my_model_func, device, train_loader, val_loader, num_epochs=20)

# Evaluate the model
conf_matrix, accuracy, precision, recall, f1 = evaluate(model, test_loader, num_classes=10, device=device, return_extra_metrics=True)

# Plot confusion matrix
ax, fig = plt.subplots(1,1, figsize=(7,5))
cm = ax.imshow(conf_matrix, cmap='viridis')
fig.colorbar(cm, ax=ax)

print(f"Test Accuracy: {accuracy}")

# Count the number of trainable parameters
num_params = count_parameters_torch(model)
print(f'The model has {num_params} trainable parameters.')

torch_models.py

This repository contains Python code for implementing various neural network architectures using PyTorch, including standard Multi-Layer Perceptrons (MLP), Convolutional Neural Networks (CNN), and K-Nearest Neighbor (KAN) layers. The code also includes a modified CNN architecture with KAN layers integrated.

Functions and Classes

1. MLP

Implements a standard Multi-Layer Perceptron (MLP) with configurable hidden layers and neurons.

• Parameters:

  • neurons_ls: List of integers representing the number of neurons in each layer, including the input and output layers.

• Example Usage:

  model = MLP([784, 128, 64, 10])
  

2. TestCNN

Implements a standard Convolutional Neural Network (CNN) for image classification tasks.

• Parameters:

  • num_classes: Number of output classes (default: 10).
  • num_channels: Number of input channels (default: 3).

• Example Usage:

  model = TestCNN(num_classes=10, num_channels=3)
  

3. TestCNNKAN

Implements a modified CNN with a KAN layer for image classification tasks.

• Parameters:

  • num_classes: Number of output classes (default: 10).
  • num_channels: Number of input channels (default: 3).

• Example Usage:

  model = TestCNNKAN(num_classes=10, num_channels=3)
  

4. CNN

Implements a generalized CNN for any number of convolutional layers followed by linear layers.

• Parameters:

  • img_size: Tuple containing the height and width of the input images.
  • in_channels: Number of channels in the input data.
  • num_classes: Number of output classes.
  • conv_layers: List of tuples containing the number of out_channels, kernel_size, and stride for each convolutional layer.

• Example Usage:

  model = CNN((32, 32), 3, 10, [(32, 3, 1), (64, 3, 1)])
  

5. KCNN

Implements a modified CNN with KAN layers for image classification tasks.

• Parameters:

  • img_size: Tuple containing the height and width of the input images.
  • in_channels: Number of channels in the input data.
  • num_classes: Number of output classes.
  • conv_layers: List of tuples containing the number of out_channels, kernel_size, and stride for each convolutional layer.

• Example Usage:

  model = KCNN((32, 32), 3, 10, [(32, 3, 1), (64, 3, 1)])
  

Example Workflow

Here is an example workflow using the provided classes and functions to train and evaluate a model:

1. Import Necessary Libraries:

   import torch
   from torch import nn, optim
   from torch.utils.data import DataLoader, TensorDataset
   

from oscars_toolbox.torch_models import MLP, CNN from oscars_toolbox.implement_torch import train_only, evaluate

2. **Prepare Data Loaders**:
```python
# Example data
train_data = TensorDataset(torch.randn(100, 3, 32, 32), torch.randint(0, 10, (100,)))
val_data = TensorDataset(torch.randn(20, 3, 32, 32), torch.randint(0, 10, (20,)))
test_data = TensorDataset(torch.randn(20, 3, 32, 32), torch.randint(0, 10, (20,)))

train_loader = DataLoader(train_data, batch_size=10)
val_loader = DataLoader(val_data, batch_size=10)
test_loader = DataLoader(test_data, batch_size=10)

3. Define Model and Training Parameters:

   device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
   neurons_ls = [784, 128, 64, 10]  # Example for MLP
   conv_layers = [(32, 3, 1), (64, 3, 1)]  # Example for CNN
   

4. Train the Model:

   # Using MLP
   model = MLP(neurons_ls)
   trained_model, train_acc, val_acc, train_acc_ls, val_acc_ls = train_only(model, device, train_loader, val_loader)
   
   # Using CNN
   model = CNN((32, 32), 3, 10, conv_layers)
   trained_model, train_acc, val_acc, train_acc_ls, val_acc_ls = train_only(model, device, train_loader, val_loader)
   

5. Evaluate the Model:

   conf_matrix, accuracy, precision, recall, f1 = evaluate(trained_model, test_loader, num_classes=10, device=device, return_extra_metrics=True)
   print(f'Confusion Matrix:\n{conf_matrix}')
   print(f'Test Accuracy: {accuracy}')
   print(f'Test Precision: {precision}')
   print(f'Test Recall: {recall}')
   print(f'Test F1 Score: {f1}')
   

6. Count Trainable Parameters:

   num_params = count_parameters_torch(trained_model)
   print(f'The model has {num_params} trainable parameters.')