amgd 1.3.3

Adaptive Momentum Gradient Descent for Regularized Poisson Regression

AMGD: Adaptive Momentum Gradient Descent

A Python implementation of Adaptive Momentum Gradient Descent (AMGD) for high-dimensional sparse Poisson regression with L1 and Elastic Net regularization. AMGD combines adaptive learning rates with momentum-based updates and specialized soft-thresholding to achieve superior performance in feature selection and optimization efficiency.

🚀 Key Features

• Optimization for Poisson regression with automatic feature selection
• Superior convergence compared to Adam, AdaGrad, and GLMnet
• Adaptive soft-thresholding for effective L1 and Elastic Net regularization
• High-dimensional support with excellent scalability (tested up to 1000+ features)
• Built-in benchmarking tools for algorithm comparison
• Extensible framework supporting custom penalties and GLM families
• Comprehensive visualization for convergence analysis and coefficient paths

📊 Performance Highlights

Based on extensive empirical evaluation:

• 38% faster convergence than Adam on average
• 27% better feature selection precision compared to standard methods
• Improved sparsity while maintaining predictive accuracy
• Robust performance across different noise levels and data dimensions

📦 Installation

From PyPI

pip install amgd

From Source

git clone https://github.com/yourusername/amgd.git 
cd amgd
pip install -e .

⚡ Quick Start

Basic Poisson Regression

from amgd.models import PoissonRegressor
from sklearn.model_selection import train_test_split

# Prepare your data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Create and train model
model = PoissonRegressor(
    optimizer='amgd',
    penalty='l1',
    lambda1=0.1,
    max_iter=1000
)
model.fit(X_train, y_train)

# Make predictions
y_pred = model.predict(X_test)

# Evaluate
score = model.score(X_test, y_test)
print(f"Test score: {score:.4f}")

Use Elastic Net for Grouped Feature Selection

model = PoissonRegressor(
    optimizer='amgd',
    penalty='elasticnet',
    lambda1=0.05,  # L1 penalty
    lambda2=0.05,  # L2 penalty
    max_iter=1000
)
model.fit(X_train, y_train)

# Check sparsity
sparsity = 1 - (np.sum(model.coef_ != 0) / len(model.coef_))
print(f"Sparsity: {sparsity:.2%}")

📖 Comprehensive Examples

1. Ecological Health Analysis

from amgd.benchmarks.datasets import load_ecological_dataset
from amgd.models import PoissonRegressor
from amgd.visualization import plot_coefficient_path

# Load ecological dataset
X, y, feature_names = load_ecological_dataset()

# Hyperparameter tuning
lambda_values = np.logspace(-4, 1, 50)
best_score = float('inf')
best_lambda = None

for lambda_val in lambda_values:
    model = PoissonRegressor(
        optimizer='amgd',
        penalty='l1',
        lambda1=lambda_val
    )
    model.fit(X_train, y_train)
    val_score = -model.score(X_val, y_val)

    if val_score < best_score:
        best_score = val_score
        best_lambda = lambda_val

# Train final model
final_model = PoissonRegressor(
    optimizer='amgd',
    penalty='l1',
    lambda1=best_lambda,
    max_iter=1000
)
final_model.fit(X_train, y_train)

# Visualize coefficient paths
plot_coefficient_path(
    lambda_values,
    coefficients,
    feature_names=feature_names,
    title="Coefficient Paths for Biodiversity Prediction"
)

2. Algorithm Comparison

from amgd.benchmarks import compare_optimizers

results = compare_optimizers(
    X, y,
    optimizers=['amgd', 'adam', 'adagrad'],
    penalties=['l1', 'elasticnet'],
    cv_folds=5,
    test_size=0.2
)

print(results['test_results'])

3. Custom Regularization

from amgd.core.penalties import PenaltyBase
from amgd.models import GLM

class AdaptiveLassoPenalty(PenaltyBase):
    def __init__(self, lambda1, weights):
        self.lambda1 = lambda1
        self.weights = weights

    def __call__(self, x):
        return self.lambda1 * np.sum(self.weights * np.abs(x))

    def proximal_operator(self, x, step_size):
        threshold = self.lambda1 * self.weights * step_size
        return np.sign(x) * np.maximum(np.abs(x) - threshold, 0)

model = GLM(
    optimizer='amgd',
    family='poisson',
    link='log'
)

🧮 Algorithm Details

AMGD integrates three key methods:

1. Adaptive Learning Rate Decay

αₜ = α / (1 + ηt)

2. Momentum Updates with Bias Correction

mₜ = ζ₁mₜ₋₁ + (1 - ζ₁)∇f(βₜ)
vₜ = ζ₂vₜ₋₁ + (1 - ζ₂)(∇f(βₜ))²
m̂ₜ = mₜ / (1 - ζ₁ᵗ)
v̂ₜ = vₜ / (1 - ζ₂ᵗ)

3. Adaptive Soft-Thresholding

βⱼ ← sign(βⱼ) · max(|βⱼ| - αₜλ/(|βⱼ| + ε), 0)

🛠️ Advanced Features

Warm Starting

model = PoissonRegressor(
    optimizer='amgd',
    warm_start=True,
    max_iter=100
)
model.fit(X_train, y_train)
model.max_iter = 500
model.fit(X_train, y_train)

Custom Convergence Criteria

from amgd.core.convergence import RelativeChangeCriterion

criterion = RelativeChangeCriterion(tol=1e-8, patience=5)
optimizer = AMGDOptimizer(convergence_criterion=criterion)

Visualization Tools

from amgd.visualization import plot_convergence, plot_feature_importance

plot_convergence(model.loss_history_, log_scale=True)
plot_feature_importance(
    model.coef_,
    feature_names=feature_names,
    top_k=20
)

🔧 API Reference

Core Classes

• AMGDOptimizer: Main optimization algorithm
• PoissonRegressor: Poisson regression with AMGD
• GLM: General framework for GLMs with various families

Key Parameters

Parameter	Default	Description
alpha	0.01	Initial learning rate
beta1	0.9	First moment decay rate
beta2	0.999	Second moment decay rate
lambda1	0.0	L1 regularization strength
lambda2	0.0	L2 regularization strength
T	20.0	Gradient clipping threshold
eta	0.0001	Learning rate decay factor

📚 Documentation

Full documentation is available at https://amgd.readthedocs.io

• Installation Guide
• Tutorial
• API Reference
• Theory & Algorithm

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

# Run tests
pytest tests/

# Run linting
flake8 src/

# Build documentation
cd docs && make html

📝 Citation

If you use AMGD in your research, please cite our paper:

@article{bakari2025amgd,
  title={Adaptive Momentum Gradient Descent Algorithm: A New Algorithm in Regularized Poisson Regression},
  author={Bakari, Ibrahim and Özkale, M. Revan},
  journal={Journal Name},
  year={2025}
  volume={XX},
  number={X},
  pages={XXX-XXX},
  doi={10.XXXX/XXXXX}
}

📄 License

This project is licensed under the MIT License

🙏 Acknowledgments

• Inspired by advances in adaptive optimization methods
• Built on top of NumPy, SciPy, and scikit-learn
• Çukurova University - Department of Statistics
• Research Community - For valuable feedback and suggestions

📧 Contact

Author: Ibrahim Bakari
Email: [e-mail]
Issues: GitHub Issues