metawalker

Learned black-box optimizers for big, complex search spaces, packaged as a drop-in replacement for scikit-learn's GridSearchCV / RandomizedSearchCV and as a plain minimize() function.

The optimizers were trained once, with evolution strategies, on cheap synthetic problems. After that they run on numpy alone, and each policy is a few KB.

pip install "metawalker[sklearn]"        # from the repo: pip install -e ".[sklearn]"

Use it

from metawalker import minimize, Float, Int, Categorical

space = {"lr": Float(1e-5, 1e-1, log=True), "layers": Int(1, 8), "act": ["relu", "tanh"], "dropout": (0.0, 0.5)}
result = minimize(train_and_score, space, budget=300, method="auto", maximize=True, recheck=3)
result.best_params, result.best_value, result.recheck_values

As a scikit-learn search estimator (it uses the same internal hook as GridSearchCV, so cv, scoring, refit, n_jobs, cv_results_, best_params_ and best_estimator_ all behave the same):

from metawalker.sklearn import MetaWalkerCV

search = MetaWalkerCV(model, {"C": Float(1e-3, 1e3, log=True), "kernel": ["rbf", "linear"]},
                     n_iter=100, method="rlcma", cv=5).fit(X, y)

A GridSearchCV param_grid (a dict of value lists) works unchanged as param_space.

Choose your own panel: test methods on your own losses

You know your problem best, so the package helps you measure which method fits it. Give compare() your own loss functions, and optionally some simulated stand-ins shaped like your problem. It runs every method several times on each, with the same budget, and reports average rank, head-to-head win rates and time per run:

from metawalker import compare, make_problem, minimize

res = compare({"my_loss": (my_fn, my_space),
               "stand-in": make_problem("plateau", n=80, noise=0.01)},  # sphere, ellipsoid, rastrigin,
              methods=["rlcma", "walker_hp", "cmaes", "gp"],          # rosenbrock, ackley, levy, plateau
              budget=200, seeds=5)
print(res.summary())

minimize(my_fn, my_space, budget=600, method=["rlcma", "gp"])   # then run the panel you picked

The simulated problems come from the same generator as our benchmark: rotated and shifted, with a known optimum, so compare() scores them by true regret rather than by lucky noisy readings.

What else it does for you

import metawalker as mw

# 1. don't know your landscape? probe it (~8% of the budget), then let it pick from the benchmark's rules
res = mw.minimize_auto(my_fn, my_space, budget=500)
print(res.recommendation.reasons)   # e.g. "92% of probes returned the same value: a plateau objective -> walker_hp"

# 2. a best result you can trust: re-test the top candidates on fresh evaluations, pick by mean, report a CI
cert = mw.certify(my_fn, res, top_k=5, repeats=5)
print(cert.summary())               # cert.shrinkage = how much the raw best score was inflated by noise

# 3. which parameters mattered, which direction helps, and the best range (free, from the search history)
print(mw.importance(res).summary())

# 4. train an optimizer specialized to YOUR kind of problem, on cheap simulated stand-ins
policy = mw.train([lambda: my_simulator_factory()], init="walker_hp", generations=50, n_jobs=8)
policy.save("my_optimizer.npz"); policy.minimize(my_fn, my_space, budget=300)

# 5. evaluate on your own workers / cluster, or plug into an existing Optuna study
with mw.AskTell(my_space, budget=200, method="rlcma") as at:
    while not at.done:
        tid, params = at.ask(); at.tell(tid, run_somewhere(params))
from metawalker.asktell import MetaWalkerSampler   # optuna.create_study(sampler=MetaWalkerSampler(dists, budget=200))

# 6. compare methods on your own losses, in parallel
mw.compare({"mine": (my_fn, my_space)}, methods=["rlcma", "gp", "cmaes"], seeds=5, n_jobs=8)

Notes:

• compare(n_jobs>1) and train(n_jobs>1) need picklable functions, plus an if __name__ == "__main__": guard on Windows.
• certify() and importance() read the search direction from the Result.
• Importance computed from an optimizer's history is biased toward the regions that optimizer sampled. Random-search histories give the cleanest estimate.

Which method

Any method also takes warmup=f. It spends that fraction of the budget on Latin-hypercube points first, and those points count toward the budget and appear in the history. The walkers then start from the best 4 points and the CMA-based methods from the best one, while the other methods simply run on what is left. Use it when the signal is rare: without it the walkers shrink and restart instead of exploring, so plain random search finds a tiny live region more often than they do. The distilled rules come from the policy study (results/analysis/policy_report.md). The numbers above are medians over 3 seeds at budget 1000, from sphere, Rosenbrock, Rastrigin, Ackley, Levy and plateau at 10 and 50 parameters. compare() accepts all of these methods.

Benchmark scope. Rotated and shifted synthetic functions (sphere, ellipsoid, Rastrigin, Rosenbrock, plus held-out Ackley and Levy) with and without noise, a honeypot-like plateau family, and a real RL honeypot search. Budgets were 300–10,000 evaluations. Each learned method was trained 3 times from scratch, with results reported per training run. A comparison against Optuna GP, Optuna multivariate TPE, SMAC3 and Nevergrad NGOpt is in progress.

Behavior to know

• Invalid configurations: if your function raises or returns nan/inf, that counts as a failed trial (result.n_failed). The search keeps going.
• Noisy objectives: the best observed value is biased upward (the winner's curse). Use recheck=k to re-evaluate the winner k more times.
• Categorical parameters are one coordinate split into equal bins. That's fine for a few choices, but unordered categoricals with many choices are a known weak spot.
• 1-parameter searches with the CMA-based methods add an ignored dummy coordinate, because pycma needs 2+.
• The panel budget is split equally, with no racing between methods.

Tests

pytest from the repo root. tests/test_regression.py checks that every optimizer reproduces the pre-refactor benchmark code bit for bit (tests/golden.json).