pylafs 0.1.0

PyLAFS — Learning Attention-based Feature Selection: a PyTorch-powered, scikit-learn compatible feature selector using trainable attention.

PyLAFS — Learning Attention-based Feature Selection

A PyTorch-powered, scikit-learn compatible feature selection method that uses a trainable attention mechanism to rank and select the most informative features for classification tasks.

Installation

pip install pylafs

Or in development/editable mode:

pip install -e .

Quick Start

from pylafs import LAFSSelector
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import pandas as pd

# Generate sample data
X, y = make_classification(n_samples=1000, n_features=50,
                           n_informative=10, random_state=42)
X = pd.DataFrame(X, columns=[f"f_{i}" for i in range(50)])
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Scale features
scaler = StandardScaler()
X_train = pd.DataFrame(scaler.fit_transform(X_train), columns=X.columns)
X_test  = pd.DataFrame(scaler.transform(X_test), columns=X.columns)

# Select top-10 features with LAFS
selector = LAFSSelector(k=10, epochs=50, lr=0.001, lambda_reg=0.01)
selector.fit(X_train, y_train)

X_train_selected = selector.transform(X_train)
X_test_selected  = selector.transform(X_test)

print("Selected features:", selector.selected_features_)
print("Feature importances:", selector.feature_importances_)

API Reference

LAFSSelector

Parameter	Type	Default	Description
k	int	10	Number of features to select
lr	float	0.001	Adam learning rate
lambda_reg	float	0.01	Entropy regularisation coefficient
epochs	int	50	Training epochs
batch_size	int	256	Mini-batch size
attn_hidden_size	int	128	Attention network hidden layer width
attn_dropout_rate	float	0.5	Dropout rate in attention network
cls_hidden_size	int	128	Classifier hidden layer width
weight_decay	float	1e-4	L2 regularisation for Adam
device	str/None	None	"cuda", "cpu", or auto-detect
random_state	int/None	None	Seed for reproducibility
verbose	bool	False	Print training progress

Methods

Method	Description
fit(X, y)	Train the LAFS model and compute feature importances
transform(X)	Reduce X to the selected features
fit_transform(X, y)	Fit and transform in one step
get_support(indices=False)	Boolean mask or indices of selected features
get_feature_names_out()	Names of the selected features

Attributes (available after fit)

Attribute	Description
feature_importances_	Mean attention weight per feature (numpy array)
ranking_	Feature indices sorted by importance (descending)
selected_features_	List of selected feature names
selected_indices_	Array of selected feature indices
model_	The trained PyTorch LAFSModel

How It Works

LAFS jointly trains two neural networks:

1. AttnNet — produces per-feature softmax attention weights
2. ClassNet — a feedforward classifier that operates on attention-weighted features

The loss function combines cross-entropy classification loss with an entropy regularisation term on the attention weights. This encourages the attention to focus on the most discriminative features.

After training, global feature importance is computed as the mean attention weight across all training samples. The top-k features by importance are selected.

License

MIT