Adaptive Greedy Search (AGS)
Adaptive Greedy Search (AGS) v3.3.0 is a surrogate-guided hyperparameter optimizer for finite, discrete search spaces. It combines a shared surrogate model, a swarm of local-search climbers, space-filling initialization, and optional fold-level cross-validation pruning.
View the v3.3.0 release · Changelog
from ags import AdaptiveGreedySearch
How AGS works
AGS is intended for finite hyperparameter grids that are too expensive to exhaustively evaluate, but where nearby configurations may have related performance.
- Typed search-space geometry. Numerical dimensions use normalized value-space distance (or log-distance when requested), ordinal dimensions use rank distance, and nominal categories use Hamming distance and one-hot features. Existing list-based grids remain supported.
- A shared surrogate. Climbers share one model trained on completed, full-CV evaluations. Random Forest is the default for mixed and discrete spaces; Gaussian Process (GP) and KDE density-ratio alternatives are also available.
- Space-filling warm-up. AGS starts with Max-Min / farthest-point seeding and keeps using space-filling proposals until the configured number of complete observations is available.
- Multiple local-search climbers. Each active climber proposes an unevaluated neighbor using the acquisition scores. Proposals are deduplicated before a batch is evaluated in parallel with
joblib. - Plateau escape and restart. Climbers can search outward through nearby rings when boxed in, and retired climbers can be respawned at promising unexplored states.
- Optional fold-level pruning. A candidate can stop before all CV folds are run when it is out of contention under the selected pruning rule. A pruned partial mean is never used as the official best score or as a complete surrogate observation.
Unlike exhaustive grid search, AGS does not guarantee that it will find the global best configuration. Its advantage depends on the evaluation budget, the structure of the performance landscape, the estimator's cost, and the selected pruning strategy.
AGS may be a poor fit when performance has little relationship to search-space proximity, the grid is very small, the budget is too small to support useful warm-up and local search, or you require an exhaustive optimum guarantee. AGS currently operates on a finite, materialized grid; conditional search spaces and lazy candidate generation are future roadmap work.
Installation
pip install adaptive-greedy-search
Or install from a local clone:
git clone https://github.com/MJwd-Hasan/adaptive-greedy-search.git
cd adaptive-greedy-search
pip install -e .
Requires Python 3.8+ and installs numpy, scikit-learn, and joblib automatically.
Quickstart
This example tunes a scikit-learn classifier on a finite grid:
from ags import AdaptiveGreedySearch
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
X, y = make_classification(
n_samples=500,
n_features=20,
random_state=0,
)
param_grid = {
"n_estimators": [50, 100, 150, 200],
"max_depth": [3, 5, 7, 9, None],
"min_samples_split": [2, 4, 6, 8],
}
search = AdaptiveGreedySearch(
estimator=RandomForestClassifier(random_state=0),
param_grid=param_grid,
cv=5,
scoring="accuracy",
max_evaluations=30,
n_climbers=6,
n_jobs=-1,
random_state=42,
)
search.fit(X, y)
print("Best parameters:", search.best_params)
print("Best CV score:", search.best_score)
print(
"Folds saved:",
f"{search.folds_saved}/{search.total_folds_possible}",
)
print(
"Climbers spawned:", search.climbers_spawned,
"| merged:", search.climbers_merged,
"| plateau escapes:", search.plateau_escapes,
)
Tip: Choose n_climbers with the evaluation budget in mind. When left as None, AGS creates about 75% of the available CPU cores as climbers (at least one); this can be too many for a very small budget.
Defining the search space
Plain Python lists are still supported:
param_grid = {
"C": [0.01, 0.1, 1, 10],
"kernel": ["linear", "rbf"],
}
AGS automatically infers pure numeric lists as numerical parameters and other lists as categorical parameters. You can specify a type explicitly when its geometry matters:
from ags import (
AdaptiveGreedySearch,
CategoricalParameter,
NumericParameter,
OrdinalParameter,
)
param_grid = {
"C": NumericParameter([0.01, 0.1, 1, 10], scale="log"),
"kernel": CategoricalParameter(["linear", "rbf"]),
"degree": OrdinalParameter([2, 3, 4, 5]),
}
The example above uses an SVC-style parameter space. The parameter types behave as follows:
| Type | Geometry | Use it for |
|---|---|---|
NumericParameter(values) |
Normalized distance between numerical values | Integer/float grids where numeric gaps matter |
NumericParameter(values, scale="log") |
Normalized distance in log space; all values must be positive | Log-spaced quantities such as C or learning_rate |
OrdinalParameter(values) |
Normalized rank distance | Ordered choices where rank matters more than numeric gap |
CategoricalParameter(values) |
Different categories have distance 1; one-hot surrogate features | Nominal choices such as kernels or split criteria |
All parameter values must be finite and explicitly listed. A parameter type does not create a continuous search interval. Conditional parameters are not yet supported, so AGS forms the Cartesian product of the provided values.
Surrogates and acquisition functions
Version 3.3 separates the surrogate model from the acquisition rule that scores candidate configurations.
Surrogate choices
surrogate_type |
Meaning | Notes |
|---|---|---|
"rf" (default) |
Random Forest regression surrogate | Suitable default for mixed/discrete grids. Between-tree dispersion is an exploration proxy, not calibrated posterior uncertainty. |
"gp" |
Gaussian Process surrogate | Provides predictive mean and standard deviation; often useful for smaller numerical spaces. |
"kde" |
KDE density-ratio scoring | Ranks candidates by log-density ratio of good versus bad observations; it is not a predictive mean/std model. |
"tpe" |
Deprecated compatibility alias for "kde" |
Kept for compatibility; this is not a full Tree-structured Parzen Estimator implementation. |
Acquisition choices
acquisition |
Behavior | Compatible surrogates |
|---|---|---|
"auto" (default) |
Uses UCB for RF/GP and density-ratio scoring for KDE | RF, GP, KDE |
"ucb" |
Upper Confidence Bound-style score: predicted mean + exploration_weight × spread |
RF, GP |
"ei" or "expected_improvement" |
Expected Improvement ranking | RF, GP |
"density_ratio" |
Uses the KDE good/bad log-density ratio | KDE only |
For Random Forest, both UCB's spread and the EI calculation rely on between-tree dispersion, so these rankings are approximations rather than calibrated Bayesian uncertainty calculations. With GP, Expected Improvement uses the GP predictive distribution.
Example:
search = AdaptiveGreedySearch(
estimator,
param_grid,
surrogate_type="rf",
acquisition="ei",
min_observations=12,
)
Surrogate warm-up
AGS does not use surrogate-guided proposals immediately. By default, it waits for:
max(2 * number_of_parameters, 10)
complete CV evaluations, capped by the number of states in the finite grid. Before then, it uses space-filling proposals. Set min_observations to override the target. Warm-up counts completed evaluations, not pruned partial runs.
Cross-validation pruning
AGS can stop evaluating a candidate before running all folds. This can save estimator fits, but pruning behavior depends on the selected rule.
pruning_strategy |
Behavior | Important caveat |
|---|---|---|
"percentile" (default) |
Compares a candidate's partial CV mean with historical partial means at the same fold index | Heuristic; not a guarantee of preserving the best configuration |
"optimistic" |
Bounds the best possible final mean using a score upper bound | Supply a valid score_upper_bound; otherwise AGS disables optimistic pruning and warns |
"none" |
Runs every CV fold for each selected configuration | Useful for baselines or debugging |
For example, accuracy has a valid upper bound of 1.0:
search = AdaptiveGreedySearch(
estimator,
param_grid,
scoring="accuracy",
pruning_strategy="optimistic",
score_upper_bound=1.0,
)
Only use "optimistic" when the bound you supply is genuinely valid for the scorer. Do not assume that 1.0 is a valid upper bound for every scikit-learn scoring metric. For a full-CV baseline without fold pruning, select pruning_strategy="none".
Parallelism notes
Candidate configurations in a batch are evaluated concurrently via joblib. Each worker sees the partial-history snapshot taken before that batch was dispatched, so candidates in the same batch cannot use one another's newly produced partial folds for percentile pruning. This slightly softens percentile pruning within a batch.
Parallel execution is workload-dependent: for cheap folds or a single-core machine, process overhead can make n_jobs > 1 slower than n_jobs=1. Also consider limiting the estimator's own threads; AGS sets the estimator's n_jobs to 1 when that parameter exists, to reduce nested parallelism.
Key parameters
| Parameter | Default | Description |
|---|---|---|
cv |
5 |
Number of CV folds. |
scoring |
"accuracy" |
Scikit-learn scorer name, such as "accuracy", "roc_auc", or "neg_mean_squared_error". |
n_climbers |
None → about 75% of CPU cores, minimum 1 |
Number of search climbers. Consider setting this explicitly for small evaluation budgets. |
climber_patience |
3 |
Consecutive non-improving steps before a climber is retired as stagnated. |
n_jobs |
-1 |
Number of parallel workers for candidate evaluation. |
max_evaluations |
25 |
Hard cap on attempted grid states, bounded by grid size. |
surrogate_type |
"rf" |
"rf", "gp", "kde", or deprecated alias "tpe". |
acquisition |
"auto" |
"auto", "ucb", "ei" / "expected_improvement", or "density_ratio" when compatible with the selected surrogate. |
min_observations |
None |
Warm-up target; defaults to max(2 * parameter_count, 10), capped by grid size. |
rf_n_estimators |
200 |
Number of trees for the Random Forest surrogate (minimum 2). |
exploration_weight |
2.0 |
Weight applied to predictive spread by UCB. |
pruning_strategy |
"percentile" |
"percentile", "optimistic", or "none". |
pruning_percentile |
25 |
Percentile threshold used by heuristic pruning; higher values generally prune more aggressively. |
max_neighbor_radius |
4 |
Maximum ring radius used when a climber escapes a plateau. |
early_stopping_patience |
None |
Optional global stop after this many evaluations without improving the best score. |
Results and evaluation semantics
After search.fit(X, y), the main result attributes include:
best_params,best_score,best_state: the best configuration with a complete CV evaluation.history: one record per attempted candidate, includingn_folds_used,n_folds_total,climber_id, and pruning status/reason.n_evaluations,n_completed_evaluations,n_partial_evaluations,total_time,stopped_early: run summary.n_pruned,total_folds_run,total_folds_possible,folds_saved: cross-validation work and pruning summary.climbers,climbers_spawned,climbers_merged,plateau_escapes: information about swarm behavior.completed_evaluationsandpartial_evaluations: separate stores for fully evaluated and incomplete/pruned results.
AGS trains its surrogate using completed CV observations only. Partial CV results can appear in history for diagnostics but cannot become the reported winner.
License
MIT © Mohammad Jawad Hasan
Metadata
Release files for adaptive-greedy-search 3.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| adaptive_greedy_search-3.3.0.tar.gz | 25.1 kB | With provenance | Details for adaptive_greedy_search-3.3.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| adaptive_greedy_search-3.3.0-py3-none-any.whl | Python 3 | none | any | Details for adaptive_greedy_search-3.3.0-py3-none-any.whl |
Total release size: 48.4 kB
Release files / adaptive_greedy_search-3.3.0.tar.gz
| Download URL | adaptive_greedy_search-3.3.0.tar.gz |
|---|---|
| Size | 25.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
49f9139f658e0f5a7755693a9fd0c8a41498b8eb537ea3d2973ba1b75c44419d
|
|
BLAKE2b-256 checksum How to use checksums |
65940b708ca5a0914cfc49de11ff00c00bab78e714af3c4c0a6c25dded0b0576
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.
Transparency logRelease files / adaptive_greedy_search-3.3.0-py3-none-any.whl
| Download URL | adaptive_greedy_search-3.3.0-py3-none-any.whl |
|---|---|
| Size | 23.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
fac46b23b2f0fd6ad4cea242d670aac5a34b9fe4d930985dcd0a17911651a6e2
|
|
BLAKE2b-256 checksum How to use checksums |
3db329f471f94008b75c131cd7e098ded16844b35b24aaa316bcbc09a38d2bec
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.
Transparency logSecurity for adaptive-greedy-search 3.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for adaptive_greedy_search-3.3.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.