Machine learning pipeline for accelerators
Project description
ML Pipeline
Machine learning pipeline.
How to create and run an Agent?
To create an Agent you need to clone the ml-pipe-service repository and install ml-pipe-core.
git clone
mkdir .venv
pipenv install .
Afterwards you create a class which inherits form SimpleService.
class SillyAgent(SimpleService):
def __init__(self, name, adapter):
super().__init__(name, adapter)
self.factor = 1.0
def reconfig_event(self, msg):
factor = msg.get('factor')
if factor is None:
_logger.error("reconfig message doesn't have 'factor' as a key.")
else:
_logger.debug(f"set new factor to: {factor}")
self.factor = factor
def proposal(self, params):
_logger.debug(f"use factor: {self.factor}")
hcors = self.machine_adapter.get_hcors(names=self.machine_adapter.get_hcor_device_names(), is_group_call=True)
self.set_machine(SetMachineMessage({'hcor': {"names": self.machine_adapter.get_hcor_device_names(), "values": [val * self.factor for val in hcors]}}))
if __name__ == '__main__':
agent = SillyAgent('silly_agent', PetraSimulationAdapter())
agent.init_local()
Inside this class you need to overload
def proposal(self, params)
which will be called when the Agent is triggered. The Machine/Simulation can be set with
def set_machine(self, data: SetMachineMessage)
This function is taking an SetMachineMessage as input parameter. SetMachineMessage holds a python dictionary with which you can set the Machine/Simulation. Currently only setting horizontal/vertical correctors, machine parameters and twiss parameters are supported. The structure of the dictionary have to look like this:
{'hcor': {'name': str, 'values': List[float]},
'vcor': {'name': str, 'values': List[float]},
'machineparams': {'name': str, 'values': List[float]},
'twiss': {'name': str, 'mat': List[List[float]]} # only for simulation
}
Optionally you can overload
def reconfig_event(self, msg)
which give you the possibility to reset internal parameters without restarting your Agent.
After creating an Agent you need to configure the environments variables to tell the Agent where to the server.
mkdir .env
Here is an example of .env file.
Notes: You have to replace localhost with the real server name.
PYTHONPATH=.
EXTERNAL_HOST_ADDR=localhost
PIPE_KAFKA_BROKERS=localhost:9094,localhost:9096
PIPE_SIMULATION_DB=localhost:3306
SIM_SQL_USER=xxxxx
SIM_SQL_PW=xxxxx
OBSERVER_URL=localhost:5003
REGISTER_URL=localhost:5004
To run the Agent you need to load the .env file. This can be done by activate the python virtual environment with pepenv.
pipenv shell
Afterwards you can run the Agent
python agents/testing/silly_agent.py
To interact with the pipeline (e.g trigger, start, list, stop Agents) you can used KafkaPipeClient.
from ml_pipe_core.client.kafka_client import KafkaPipeClient
pipe = KafkaPipeClient()
print(pipe.get_running_service_info())
p_id = pipe.run_service('SillyAgent')
pipe.get_observation(p_id) # wait for response
An GUI will be come soon.
Developer: Core Services and ml_pipe_core Package
The ML-Pipeline is using docker to run the infrastructure and test it locally. For this you need docker and docker-compose >= 1.28.0.
Build Docker images
To build all docker images run:
make build-all
This can take a while...
Notes: After changing code you just need to rebuild the service images, which is much faster.
make build-service-images
Start Core Services
To Start the core services of the ml-pipeline you just need to execute
make up ENV_FILE=.env
An .env file is needed to configure the ml-pipeline. Here is a .env file for a local configuration.
PYTHONPATH=.
EXTERNAL_HOST_ADDR=localhost
PIPE_KAFKA_BROKERS = kafka_1:9092,kafka_2:9096
PIPE_SIMULATION_DB = simulation_database:3306
OBSERVER_URL = observer:5003
REGISTER_URL = register:5004
SIM_SQL_USER=xxxx
SIM_SQL_PW=xxxx
PATH_TO_ML_PIPE_SERVICES_ROOT=/path/to/ml-pipe-services
In production replace EXTERNAL_HOST_ADDR=localhost with the server url and set PATH_TO_ML_PIPE_SERVICES_ROOT to the location of the cloned ml-pipe-services repository.
Stop Core Services
You have to stop the docker container at the end.
make down
Tests locally
Note: Docker and docker-compose => 2.80 is required to run tests locally
Run all tests:
make tests ENV_FILE=.env
Run end to end tests:
make e2e-tests ENV_FILE=.env
Run integration tests:
make integration-tests ENV_FILE=.env
Run unit tests
make unit-tests ENV_FILE=.env
Additional Stuff
Maxwell
Log in via ssh to max-wgs.desy.de
Install python 3.8.8
wget https://repo.anaconda.com/archive/Anaconda3-2021.05-Linux-x86_64.sh
sh Anaconda3-2021.05-Linux-x86_64.sh
export PATH=$HOME/anaconda3/bin:$PATH
PyTine and K2I2K_os on Machine PETRA III
The machine observer and controller for PETRA III are using the PetraAdapter which is using the libs PyTine and K2I2K_os. The Path to this libs have to be added to the PYTHONPATH.
For PyTine have a look at https://confluence.desy.de/display/HLC/Developing+with+Python.
K2I2K_os can be cloned via git from:
git clone https://username@stash.desy.de/scm/pihp/petra3.optics.tools.git
Jupyter notebook
To use KafkaPipeClient in a Jupyter notebook you need to add the virtual environment to Jupyter.
First activate the python virtual environment.
For Pipenv
pipenv shell
Start the jupyter notebook:
pip install --user ipykernel
Note: You have to add the virtual enviroment to jupyter. First, activate the virutal environment. Then run:
python -m ipykernel install --user --name=<myenv>
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for ml_pipe_core-0.0.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2ea225225965199e9165195d537a57b519ea81cfbd9f9381fb343da8ac99fc14 |
|
MD5 | 0cb594ab80a2c980fb336daead261ed0 |
|
BLAKE2b-256 | c368b780847b0577ccbe9cd69d994256febcd81a3f1bdb3f07dc20df7f5b2faf |