Lerna

A high-performance configuration framework for Python applications, built with Rust.

Lerna is a rewrite of Facebook's Hydra configuration framework. It provides the same powerful API with significantly improved performance through a Rust core.

Features

• Same Hydra API: Drop-in replacement for Hydra - just change import hydra to import lerna
• Rust-powered: Core config parsing and loading implemented in Rust via PyO3
• Full Compatibility: 2,854 tests passing, nearly 100% Hydra compatibility
• No ANTLR: Override parser completely rewritten in Rust (~2,400 LOC removed)
• Zero Warnings: Clean Rust codebase with no compiler warnings
• Extension Points: Rust traits for Callback, ConfigSource, Launcher, and Sweeper with Python interoperability

Installation

pip install lerna

Quick Start

import lerna
from omegaconf import DictConfig

@lerna.main(config_path="conf", config_name="config")
def my_app(cfg: DictConfig) -> None:
    print(cfg.db.driver)
    print(cfg.db.user)

if __name__ == "__main__":
    my_app()

Migration from Hydra

Lerna is a drop-in replacement for Hydra. To migrate:

1. Change Imports

# Before (Hydra)
import hydra
from hydra import compose, initialize
from hydra.core.config_store import ConfigStore

# After (Lerna)
import lerna
from lerna import compose, initialize
from lerna.core.config_store import ConfigStore

2. That's It!

All your existing configs, overrides, and patterns work unchanged:

# Same CLI interface
python my_app.py db=postgres server.port=8080

# Same multirun syntax
python my_app.py -m db=mysql,postgres server.port=8080,8081

# Same sweep functions
python my_app.py -m learning_rate=interval(0.001,0.1) batch_size=choice(16,32,64)

Compatibility Notes

What Works Identically (100%)

Feature	Status
@lerna.main() decorator	✅ Identical to @hydra.main()
compose() API	✅ Same signature and behavior
initialize() / initialize_config_dir()	✅ Same API
Config composition with defaults	✅ Full support
Override syntax (key=value, +key, ~key, key@pkg)	✅ All syntax supported
Sweep functions (choice, range, interval, glob)	✅ Full support
Cast functions (int, float, str, bool, json_str)	✅ Full support
Modifiers (shuffle, sort, tag, extend_list)	✅ Full support
Structured configs (dataclasses)	✅ Full support
Package directives (@package)	✅ Full support
Interpolations (${key}, ${oc.env:VAR})	✅ Via OmegaConf
ConfigStore	✅ Full support
Shell completion (bash, zsh, fish)	✅ Full support

Known Differences (17 edge cases)

Difference	Impact	Workaround
Zsh tilde completion	16 tests	Use full paths instead of ~ in zsh completion
Multirun completion edge case	1 test	Minor CLI completion limitation

These are shell-specific completion behaviors, not functional differences.

Hydra Issues Fixed in Lerna

Lerna addresses several long-standing Hydra issues that have been open for years:

List Modification from CLI (#1547, #2477)

Lerna adds intuitive, cross-platform list operations:

# Append items to a list
python app.py 'tags=append(new_tag)'
python app.py 'tags=append(a,b,c)'  # Multiple items

# Prepend items
python app.py 'tags=prepend(first)'

# Insert at specific index
python app.py 'tags=insert(0,first_item)'

# Remove by index
python app.py 'tags=remove_at(0)'      # Remove first
python app.py 'tags=remove_at(-1)'     # Remove last

# Remove by value
python app.py 'tags=remove_value(old_tag)'

# Clear entire list
python app.py 'tags=list_clear()'

Function	Description	Example Result
append(...)	Add items to end	[a, b] → [a, b, c]
prepend(...)	Add items to beginning	[b, c] → [a, b, c]
insert(idx, val)	Insert at index	[a, c] → [a, b, c]
remove_at(idx)	Remove by index	[a, b, c] → [b, c]
remove_value(val)	Remove first match	[a, b, c] → [a, c]
list_clear()	Clear all items	[a, b, c] → []

These functions use shell-safe syntax (quote the entire override) and work on bash, zsh, fish, PowerShell, and cmd.

No More ANTLR (#2570)

Hydra's ANTLR-based parser breaks when PYTHONOPTIMIZE=1 or PYTHONOPTIMIZE=2 is set. Lerna's Rust parser has no Python dependencies and works in all environments.

# This breaks Hydra but works with Lerna
PYTHONOPTIMIZE=2 python app.py db=postgres

Default Overrides in Decorator (#2459)

Lerna adds an overrides parameter to @lerna.main() for setting default overrides that can be overridden from CLI:

@lerna.main(
    config_path="conf",
    config_name="config",
    overrides=["db.driver=postgres", "server.port=8080"]  # Default overrides
)
def my_app(cfg: DictConfig) -> None:
    print(cfg.db.driver)  # "postgres" by default, CLI can override

# Uses decorator defaults
python app.py                        # db.driver=postgres

# CLI overrides take precedence
python app.py db.driver=mysql        # db.driver=mysql

Instantiate Lookup Without Calling (#2140)

Lerna adds _call_=False to instantiate() for importing non-callable objects (like torch.int64):

from lerna.utils import instantiate
from omegaconf import OmegaConf

# Import a non-callable object directly
cfg = OmegaConf.create({
    "_target_": "torch.int64",
    "_call_": False,  # Don't try to call it
})
dtype = instantiate(cfg)  # Returns torch.int64 directly

Backward-Compatible Plugin Discovery

Lerna discovers plugins from both lerna_plugins and hydra_plugins namespaces, enabling gradual migration:

# Both work:
# - lerna_plugins.my_plugin.MyPlugin  (new Lerna plugins)
# - hydra_plugins.my_plugin.MyPlugin  (existing Hydra plugins)

Subfolder Config Append Fix (#2935)

Hydra incorrectly treats appended defaults as relative paths when the main config is in a subfolder:

# Hydra bug: this fails because it looks for server/db/postgresql
python app.py --config-name=server/alpha +db@db_2=postgresql

# Lerna: correctly treats appended configs as absolute paths
python app.py --config-name=server/alpha +db@db_2=postgresql  # Works!

Defaults List Patching (_patch_ directive)

Hydra provides no way to remove or modify specific keys/values inherited from composed configs via the defaults list. Lerna adds a _patch_ directive that lets you apply override operations to the composed config before CLI overrides are applied.

# config.yaml
defaults:
  - some_lib/defaults    # pulls in a library config
  - _self_
  - _patch_:
    - ~unwanted_key                # delete a key
    - ~status=deprecated           # delete key only if value matches
    - items=remove_value(stale)    # remove a list item by value
    - items=remove_at(0)           # remove a list item by index
    - +new_key=injected            # add a new key
    - setting=new_value            # change a value

Key resolution rules:

Syntax	Behavior	Example
_patch_:	Bare keys auto-prefix with parent config's package	~drop_me in @pkg config → ~pkg.drop_me
_patch_@vendor:	Bare keys auto-prefix with specified package	~debug → ~vendor.debug
_here_. prefix	Explicit relative to parent package	_here_.drop_me → pkg.drop_me
_global_. prefix	Absolute path from config root	_global_.root_key → root_key

For root-level configs (no @ package), bare keys and _here_ are equivalent since the parent package is empty.

Supported operations (uses lerna's full override syntax):

Operation	Syntax	Description
Delete key	~key	Remove key from config
Conditional delete	~key=value	Remove key only if current value matches
Change value	key=value	Set key to new value
Add key	+key=value	Add new key (error if exists)
Force-add key	++key=value	Set key (create if missing)
List append	key=append(v)	Add item to end of list
List prepend	key=prepend(v)	Add item to start of list
List insert	key=insert(i,v)	Insert item at index
List remove by index	key=remove_at(i)	Remove item at index
List remove by value	key=remove_value(v)	Remove first matching item
List clear	key=list_clear()	Remove all list items

Example with packaged config:

# config.yaml — using _patch_@vendor to scope bare keys to the vendor package
defaults:
  - vendor/large_defaults@vendor
  - _self_
  - _patch_@vendor:
    - ~debug_mode           # bare key → targets vendor.debug_mode
    - items=remove_value(x) # bare key → targets vendor.items

# Multiple scoped patches can target different packages:
# - _patch_@db:
#   - ~debug
# - _patch_@server:
#   - port=9090

Nested patches: _patch_ directives in sub-configs accumulate naturally. If lib/refined.yaml has its own _patch_ that removes beta, and your root config adds _patch_@lib: to remove gamma, both patches apply — beta and gamma are both removed from the final config.

Relative Path in Defaults Fix (#2878)

Hydra produces empty string keys when using .. in defaults list paths:

# Hydra bug with ../dir2 produces config with empty string keys
# Lerna normalizes paths correctly
defaults:
  - ../dir2: child.yaml  # Now works correctly

importlib-resources 6.2+ Compatibility (#2870)

Hydra breaks with importlib-resources 6.2+ due to OrphanPath objects not having is_file()/is_dir() methods. Lerna handles this gracefully.

Plugin Registration Compatible with Hydra

Lerna provides a bridge that allows plugins registered via lerna to work with hydra-core. This enables you to write plugins once and have them work with both frameworks.

Registering Plugins via Entry Points

Add your plugin to pyproject.toml using the hydra.lernaplugins entry point group:

# For SearchPathPlugin modules:
[project.entry-points."hydra.lernaplugins"]
my-plugin = "my_package.plugin_module"

# For package-style config directories:
[project.entry-points."hydra.lernaplugins"]
my-plugin = "pkg:my_package.hydra"

# If only using lerna, you can also register under lerna.plugins:
[project.entry-points."lerna.plugins"]
my-plugin = "my_package.plugin_module"

Module-style entry points (like my_package.plugin_module) are imported and scanned for SearchPathPlugin subclasses.

Package-style entry points (like pkg:my_package.hydra) register config search paths directly.

How It Works

When hydra-core is used, lerna's LernaGenericSearchPathPlugin (installed in the hydra_plugins namespace) discovers all plugins registered under hydra.lernaplugins and makes them available to hydra's plugin system.

This enables gradual migration: you can write plugins for lerna and they'll automatically work with existing hydra-core installations.

Third-Party Plugins

Hydra's plugin ecosystem (Optuna, Ray, Submitit, etc.) references hydra internally. To use them with Lerna:

# Option 1: Import aliasing (recommended)
import lerna as hydra  # Alias for plugin compatibility

# Option 2: Use Lerna's built-in extensions
from lerna import RustBasicLauncher, RustBasicSweeper

Dependencies

Lerna requires OmegaConf (same as Hydra):

pip install lerna omegaconf

Performance

Operation	Hydra	Lerna	Speedup
YAML parsing	240μs	6.5μs	37x
Config composition	18,826μs	929μs	20x
Config load (cached)	-	2.0μs	-

Key Components

Override Parser (Rust)

The override parser is fully implemented in Rust with support for:

• All sweep types: choice(), range(), interval(), glob()
• Cast functions: int(), float(), str(), bool(), json_str()
• Modifiers: shuffle(), sort(), tag(), extend_list()
• User-defined functions via Python callbacks (with proper shadowing)
• Complex nested structures and interpolations

Config Loading (Rust + Python)

• High-performance YAML parsing in Rust
• Defaults list processing with proper package resolution
• Config merging and override application
• Full interpolation support via OmegaConf

Job Runner (Rust)

• Job context management
• Output directory computation and creation
• Config/override file serialization

Extension Points (Rust + Python)

Pluggable architecture allowing both Rust and Python implementations:

• Callback: Lifecycle hooks (on_job_start, on_job_end, on_run_start, etc.)
• ConfigSource: Config loading from file://, pkg://, structured:// sources
• Launcher: Job execution orchestration (BasicLauncher included)
• Sweeper: Parameter sweep strategies (BasicSweeper with cartesian product included)

Architecture

lerna/
├── lerna/              # Python package (Hydra API)
├── rust/               # Pure Rust core library (no Python deps)
│   └── src/
│       ├── parser/     # Override parser (2,800 LOC)
│       ├── config/     # Config loading
│       ├── omegaconf/  # OmegaConf compatibility
│       └── ...
└── src/                # PyO3 bindings

Test Status

Component	Tests	Status
Full Suite	2,854	✅ Passing
Parser	515	✅ Passing (0 xfailed)
Rust Core	229	✅ Passing
Extension Points	65	✅ Passing

Remaining Xfails (17)

All remaining xfails are known shell-specific limitations, not bugs:

• 16 zsh completion tests (tilde handling in shells)
• 1 multirun completion test (partial override parsing)

Development

# Build Rust extension
make develop

# Run tests
make test

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

This project is based on Hydra by Facebook Research.