LightEx: A Light Experiment Manager
Project description
LightEx
LightEx
is a lightweight experiment framework to create, monitor and record your machine learning experiments. Targeted to individual data scientists, researchers, small teams and, in general, resource-constrained experimentation.
Yet another experiment framework?
Systematic experimentation tools are essential for a data scientist. Unfortunately, many existing tools (kubeflow
, mlflow
, polyaxon
) are too monolithic, cloud-first, target very diverse audiences and hence spread too thin, and yet lack important dev-friendly features. Other tools cater only to a specific task , e.g., tensorboard
only handles log recording and visualization. Also, contrasting different experiment frameworks is hard: there is no standardized expt-management architecture for machine learning and most open-source frameworks are undergoing a process of adhoc requirements discovery.
Our USP:
- We study the anatomy of a ML expt-framework. Identify principal components: flexible configuration, job dispatcher, multi-logger, log visualization and querying, storage virtualization.
- This informs the modular design of
LightEx
: all the key components are included and integrated via a flexible configuration manager. By design, the components are decoupled, swappable and can be developed independently. Codebase is small, easily navigable (hopefully stays that way). - We don't re-invent the wheel. The job dispatcher builds over the docker/kubernetes ecosystem, we use
mlflow
as our primary log tracker and visualizer, and employ pythondataclass
based flexible configuration specs, both for jobs and models. - We target resource-constrained computing environments, small teams of researchers and quick dev-cycles. Though, hoping that our modular design eventually makes scaling a no-brainer.
- Read more about design here.
LightEx
setup cost is low for your existing projects:
- Add or update your logging code using
LightEx
'smulti_logger
API. Example here. - Update config instances in
config_expts.py
(mainly, the hyperparameter classHP
and run command templatecmd
)
Getting Started
pip install lightex
- In your project directory:
lightex init
This creates config_expts.py
, run_expts.py
in the project directory.
-
Install
docker
,microk8s
. Instructions here -
Spin up the backend e.g., mlflow tracker, tensorboard, etc.
- start backend services:
lightex up
- start backend services:
-
Configure and execute experiments
- Build your project docker container which contains all library dependencies, except the code and data, which are mounted separately.
- Update
config_expts.py
. See here for config schema. python run_expts.py --config-name C
Here,C
is a config instance defined inconfig_expts.py
. See here for examples.
Directory Structure
lex/
bin/
legen
leup
backend/
lex/
config/
config_backend.py
config_expts.py
Random
- A lot of it looks like setting up giant web of configuration variables.
- No optimal choice here:
json
,yaml
,jsonnet
— all formats have issues. - Using
dataclass
es, we can write complex config specs, with built-in inheritance and update features. Tiny bit of a learning curve here, but a lot of flexibility.
- No optimal choice here:
References
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.