AMGD: Adaptive Momentum Gradient Descent
A Python implementation of Adaptive Momentum Gradient Descent (AMGD) for high-dimensional sparse Poisson regression with L1 and Elastic Net regularization. AMGD combines adaptive learning rates with momentum-based updates and specialized soft-thresholding to achieve superior performance in feature selection and optimization efficiency.
🚀 Key Features
- 🎯 Optimized for Poisson regression with automatic feature selection
- ⚡ Superior convergence compared to Adam, AdaGrad, and GLMnet
- 🔧 Adaptive soft-thresholding for effective L1 and Elastic Net regularization
- 📊 High-dimensional support with excellent scalability (tested up to 1000+ features)
- 🏆 Built-in benchmarking tools for algorithm comparison
- 🔌 Extensible framework supporting custom penalties and GLM families
- 📈 Comprehensive visualization for convergence analysis and coefficient paths
📊 Performance Highlights
AMGD demonstrates effecient performance in regularized Poisson regression, as validated by comprehensive experiments on ecological count data:
| Metric | AMGD vs Adam | AMGD vs AdaGrad | AMGD vs GLMnet |
|---|---|---|---|
| MAE | 2.7% lower | 56.6% lower | 66.4% lower |
| RMSE | 2.9% lower | 49.2% lower | 59.0% lower |
| MPD | 3.0% lower | 81.0% lower | 92.4% lower |
| Sparsity | 11× higher | ∞ (from 0%) | ∞ (from 0%) |
| Runtime | 2× faster | >370× faster | 20× faster |
| Stat. Significance | p < 0.0001 (all) | ✅ Large effect sizes |
✅ Achieves 35.29% model sparsity (vs 11.76% for Adam, 0% for AdaGrad and GLMnet)
✅ Bootstrapped performance is stable across 1000 runs with tight confidence intervals
✅ Adaptive thresholding outperforms static regularization, enabling effective feature selection
✅ Handles high-dimensional data with robust convergence and minimal tuning
“AMGD provides up to 56.6% MAE reduction over AdaGrad, and outperforms Adam on every metric while selecting significantly fewer features.”
— Bakari & Özkale (2025)
📦 Installation
Quick Install from PyPI
pip install amgd
Development Installation
git clone https://github.com/elbakari01/amgd.git
cd amgd
pip install -e .
Dependencies
numpy>=1.21.0
scipy>=1.7.0
scikit-learn>=1.0.0
matplotlib>=3.3.0
seaborn>=0.11.0
⚡ Quick Start
30-Second Example
import numpy as np
from amgd.models import PoissonRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
# Generate sample data
X, y = make_regression(n_samples=1000, n_features=50, noise=0.1, random_state=42)
y = np.abs(y.astype(int)) # Convert to count data for Poisson
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Train AMGD model
model = PoissonRegressor(optimizer='amgd', penalty='l1', lambda1=0.1)
model.fit(X_train, y_train)
# Evaluate
print(f"Test score: {model.score(X_test, y_test):.4f}")
print(f"Selected features: {np.sum(model.coef_ != 0)}/{len(model.coef_)}")
Basic Poisson Regression
from amgd.models import PoissonRegressor
# Create and train model
model = PoissonRegressor(
optimizer='amgd',
penalty='l1',
lambda1=0.1,
max_iter=1000,
verbose=True
)
model.fit(X_train, y_train)
# Make predictions
y_pred = model.predict(X_test)
# Evaluate performance
from amgd.utils.metrics import poisson_deviance
deviance = poisson_deviance(y_test, y_pred)
print(f"Poisson deviance: {deviance:.4f}")
Elastic Net for Grouped Feature Selection
# Best for correlated features
model = PoissonRegressor(
optimizer='amgd',
penalty='elasticnet',
lambda1=0.05, # L1 penalty (sparsity)
lambda2=0.05, # L2 penalty (grouping)
max_iter=1000
)
model.fit(X_train, y_train)
# Analyze sparsity
sparsity = np.mean(model.coef_ == 0)
print(f"Sparsity: {sparsity:.2%}")
print(f"Non-zero coefficients: {np.sum(model.coef_ != 0)}")
📖 Comprehensive Examples
1. Real-World Example: Species Count Prediction
from amgd.models import PoissonRegressor
from amgd.visualization import plot_convergence, plot_feature_importance
import pandas as pd
# Load your ecological data
# X: environmental features (temperature, rainfall, etc.)
# y: species counts
data = pd.read_csv('ecological_data.csv')
X = data[['temperature', 'rainfall', 'elevation', 'soil_ph']].values
y = data['species_count'].values
# Cross-validation for hyperparameter tuning
from sklearn.model_selection import cross_val_score
lambda_values = np.logspace(-4, 1, 20)
cv_scores = []
for lam in lambda_values:
model = PoissonRegressor(optimizer='amgd', penalty='l1', lambda1=lam)
scores = cross_val_score(model, X, y, cv=5, scoring='neg_mean_poisson_deviance')
cv_scores.append(-scores.mean())
best_lambda = lambda_values[np.argmin(cv_scores)]
# Train final model
model = PoissonRegressor(optimizer='amgd', penalty='l1', lambda1=best_lambda)
model.fit(X, y)
# Visualize results
plot_convergence(model.loss_history_, title="AMGD Convergence")
plot_feature_importance(model.coef_, feature_names=['Temperature', 'Rainfall', 'Elevation', 'Soil pH'])
2. Algorithm Benchmarking
from amgd.benchmarks import compare_optimizers
# Compare multiple optimizers
results = compare_optimizers(
X, y,
optimizers=['amgd', 'adam', 'adagrad'],
penalties=['l1', 'elasticnet'],
cv_folds=5,
test_size=0.2,
random_state=42
)
# View results
print("Cross-Validation Results:")
print(results['cv_results'])
print("\nTest Set Results:")
print(results['test_results'])
# Statistical significance testing
from amgd.benchmarks import statistical_significance_test
stats_results = statistical_significance_test(
X, y,
optimizers=['amgd', 'adam'],
n_runs=50
)
3. Custom Penalty Functions
from amgd.core.penalties import PenaltyBase
import numpy as np
class AdaptiveLassoPenalty(PenaltyBase):
"""Adaptive Lasso with feature-specific weights."""
def __init__(self, lambda1, weights):
self.lambda1 = lambda1
self.weights = weights
def __call__(self, x):
return self.lambda1 * np.sum(self.weights * np.abs(x))
def gradient(self, x):
return self.lambda1 * self.weights * np.sign(x)
def proximal_operator(self, x, step_size):
threshold = self.lambda1 * self.weights * step_size
return np.sign(x) * np.maximum(np.abs(x) - threshold, 0)
# Use custom penalty
initial_weights = 1.0 / (np.abs(ridge_coef) + 0.1) # From ridge regression
penalty = AdaptiveLassoPenalty(lambda1=0.1, weights=initial_weights)
model = PoissonRegressor(optimizer='amgd', penalty=penalty)
model.fit(X_train, y_train)
🧮 Algorithm Details
AMGD combines three key innovations for superior performance:
1. Adaptive Learning Rate with Exponential Decay
αₜ = α₀ / (1 + ηt)
α₀: Initial learning rate (default: 0.01)η: Decay rate (default: 0.0001)t: Current iteration
2. Bias-Corrected Momentum Updates
mₜ = β₁mₜ₋₁ + (1 - β₁)∇f(θₜ) # First moment
vₜ = β₂vₜ₋₁ + (1 - β₂)(∇f(θₜ))² # Second moment
m̂ₜ = mₜ / (1 - β₁ᵗ) # Bias correction
v̂ₜ = vₜ / (1 - β₂ᵗ) # Bias correction
3. Adaptive Soft-Thresholding for Sparsity
θⱼ ← prox_λ(θⱼ - αₜ∇f(θⱼ))
where prox_λ(x) = sign(x) · max(|x| - λ, 0)
Why AMGD Works Better
| Component | Traditional Methods | AMGD Innovation | Benefit |
|---|---|---|---|
| Learning Rate | Fixed or simple decay | Adaptive + exponential decay | Better convergence |
| Momentum | Standard momentum | Bias-corrected dual momentum | Faster escape from local minima |
| Regularization | Basic soft-thresholding | Adaptive thresholding | Superior feature selection |
🛠️ Advanced Features
Model Persistence
import joblib
# Save trained model
model.fit(X_train, y_train)
joblib.dump(model, 'amgd_model.pkl')
# Load and use
loaded_model = joblib.load('amgd_model.pkl')
predictions = loaded_model.predict(X_new)
Warm Starting for Hyperparameter Tuning
model = PoissonRegressor(optimizer='amgd', warm_start=True)
# Progressive training
for lambda_val in [0.1, 0.05, 0.01]:
model.lambda1 = lambda_val
model.fit(X_train, y_train)
print(f"Lambda {lambda_val}: Score = {model.score(X_val, y_val):.4f}")
Custom Convergence Criteria
from amgd.core.convergence import RelativeChangeCriterion
# Custom convergence
criterion = RelativeChangeCriterion(tol=1e-8, patience=10)
model = PoissonRegressor(
optimizer='amgd',
convergence_criterion=criterion,
max_iter=2000
)
Advanced Visualization
from amgd.visualization import (
plot_convergence_comparison,
plot_coefficient_path,
plot_feature_importance
)
# Compare convergence of different optimizers
histories = {
'AMGD': model_amgd.loss_history_,
'Adam': model_adam.loss_history_,
'AdaGrad': model_adagrad.loss_history_
}
plot_convergence_comparison(histories)
# Coefficient regularization path
lambdas = np.logspace(-4, 1, 50)
coef_path = np.array([fit_model(lam).coef_ for lam in lambdas])
plot_coefficient_path(lambdas, coef_path, feature_names=feature_names)
📚 Use Cases
🧬 Bioinformatics
- Gene expression analysis
- Variant calling
- Protein abundance prediction
🌍 Ecology & Environmental Science
- Species abundance modeling
- Biodiversity prediction
- Climate impact assessment
📊 Marketing & Business
- Customer count prediction
- Website traffic modeling
- Demand forecasting
🏥 Healthcare
- Patient admission counts
- Disease occurrence modeling
- Treatment response analysis
🔧 API Reference
Main Classes
PoissonRegressor
Main class for Poisson regression with AMGD optimization.
PoissonRegressor(
optimizer='amgd', # Optimizer type
penalty='l1', # Regularization type
lambda1=0.0, # L1 penalty strength
lambda2=0.0, # L2 penalty strength
alpha=0.01, # Learning rate
beta1=0.9, # First moment decay
beta2=0.999, # Second moment decay
max_iter=1000, # Maximum iterations
tol=1e-6, # Convergence tolerance
fit_intercept=True, # Fit intercept term
warm_start=False, # Use previous solution
verbose=False # Print progress
)
Key Methods
| Method | Description | Returns |
|---|---|---|
fit(X, y) |
Train the model | self |
predict(X) |
Make predictions | ndarray |
score(X, y) |
Model performance | float |
get_params() |
Get parameters | dict |
set_params(**params) |
Set parameters | self |
Utility Functions
from amgd.utils.metrics import poisson_deviance, mean_absolute_error
from amgd.utils.validation import check_array, check_is_fitted
from amgd.benchmarks import compare_optimizers, statistical_significance_test
🧪 Testing & Validation
Run the test suite:
# Install test dependencies
pip install pytest pytest-cov
# Run all tests
pytest tests/ -v
# Run with coverage
pytest tests/ --cov=amgd --cov-report=html
# Run specific test categories
pytest tests/test_models.py -v # Model tests
pytest tests/test_optimizers.py -v # Optimizer tests
pytest tests/test_benchmarks.py -v # Benchmark tests
📈 Performance Tips
For Large Datasets (>10k samples)
model = PoissonRegressor(
optimizer='amgd',
alpha=0.001, # Lower learning rate
max_iter=500, # Fewer iterations
tol=1e-4 # Relaxed tolerance
)
For High-Dimensional Data (>1k features)
model = PoissonRegressor(
optimizer='amgd',
penalty='elasticnet',
lambda1=0.01, # Stronger L1 penalty
lambda2=0.001, # Light L2 penalty
beta1=0.95 # Higher momentum
)
Memory Optimization
# For memory-constrained environments
model = PoissonRegressor(
optimizer='amgd',
fit_intercept=False, # Skip intercept if not needed
warm_start=True, # Reuse computations
verbose=False # Reduce memory overhead
)
🤝 Contributing
We welcome contributions! Here's how to get started:
Development Setup
# Fork and clone the repository
git clone https://github.com/yourusername/amgd.git
cd amgd
# Create virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install in development mode
pip install -e ".[dev]"
# Install pre-commit hooks
pre-commit install
Contribution Guidelines
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Write tests for your changes
- Ensure all tests pass (
pytest tests/) - Follow PEP 8 style guide (
flake8 src/) - Commit your changes (
git commit -m 'Add amazing feature') - Push to your branch (
git push origin feature/amazing-feature) - Open a Pull Request
Areas We Need Help
- 🐛 Bug fixes and performance improvements
- 📖 Documentation and examples
- 🧪 Test coverage expansion
- 🌍 Internationalization support
- 📊 New visualization tools
📄 License
This project is licensed under the MIT License - see the LICENSE file for details.
📝 Citation
If you use AMGD in your research, please cite our paper:
@article{bakari2025amgd,
title={Adaptive Momentum Gradient Descent: A Novel Optimization Algorithm for Regularized Poisson Regression},
author={Bakari, Ibrahim and Özkale, M. Revan},
journal={arXiv preprint arXiv:2025.XXXXX},
year={2025},
note={Submitted for publication}
}
🙏 Acknowledgments
- Çukurova University - Department of Statistics
- Scientific Community - For valuable feedback and contributions
- Open Source Libraries - NumPy, SciPy, scikit-learn, matplotlib
- Beta Testers - Early adopters who helped improve the package
📊 Project Stats
👤 Who Should Use This?
AMGD is ideal for:
- 🔬 Researchers working with high-dimensional count data
- 📊 Data scientists needing feature selection in Poisson models
- ⚙️ ML engineers building scalable sparse regression pipelines
- 📈 Statisticians exploring novel regularization techniques
- 🧬 Bioinformaticians analyzing gene expression or variant data
- 🌍 Environmental scientists modeling species abundance and biodiversity
- 🏥 Healthcare analysts predicting patient counts and disease occurrence
Perfect For These Scenarios:
✅ High-dimensional data (p > n problems)
✅ Sparse feature selection required
✅ Count/rate data modeling
✅ Correlated predictors present
✅ Interpretable models needed
🚀 Roadmap
Planned enhancements for future releases:
📅 Version 1.5.0 (Q3 2025)
- 🚀 GPU acceleration via CuPy support
- 🔄 Cross-validation wrappers for easier hyperparameter tuning
- 📊 Feature importance stability plots and analysis
📅 Version 1.6.0 (Q4 2025)
- 🎯 Bayesian priors for adaptive regularization
- 🔗 Enhanced scikit-learn integration and pipeline support
- 📈 Multi-task learning extensions
📅 Version 2.0.0 (Q1 2026)
- 🌐 Distributed computing support
- 🧠 Neural network integration for deep sparse models
- 📱 Mobile deployment optimizations
🆘 Help Wanted
Vote for features or contribute:
📧 Support & Contact
- 📧 Email: acbrhmbakari@gmail.com
- 🐛 Issues: GitHub Issues
- 💬 Discussions: GitHub Discussions
- 📚 Documentation: Read the Docs
⭐ Star this repository if you find it useful! ⭐
Metadata
Release files for amgd 1.4.8
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| amgd-1.4.8-py3-none-any.whl | Python 3 | none | any | Details for amgd-1.4.8-py3-none-any.whl |
Release files / amgd-1.4.8-py3-none-any.whl
| Download URL | amgd-1.4.8-py3-none-any.whl |
|---|---|
| Size | 70.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8e7a59716ae53f2b5389821b786e70fefab1d2650cfe298a1ebff18ca53e5d0d
|
|
BLAKE2b-256 checksum How to use checksums |
aefb0a864f16789dedf623826fb2855214c7ab0fc15223276009405229de95c3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.7
|
Security for amgd 1.4.8
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.