configurun 0.1.0

PySide6 based user-interface tools to create and manage machine learning training/testing-configurations and run them automatically and/or remotely..

Configurun

Configurun is a cross-platform PySide6-based package that implements an application for managing, creating and (remotely) running python configurations. It was designed mainly with machine-learning tasks in mind, but can be used for any python script that takes arguments as an input. The editor-UI is created automatically using either an argparse.Argumentparser or python-@dataclass(es).

The Configurun-app is especially useful for scripts/experiments that require a lot of arguments to be tweaked across many experiment-runs. It also makes the process of running experiments remotely much easier by enabling the user to edit, add and schedule tasks on any running Configurun server-instance reachable via a network connection.

This package was created in tandem with pyside6-utils.

Table of contents

• Configurun
• Table of contents
• Features
  • Configuration Editor
  • Run Queue
  • Remote-processing
• Installation
• How to run?
  • Local App
  • Client App
  • Server-instance
• Option-source
  • Custom Options (@dataclass)
  • Custom Options (ArgumentParser)
  • Custom Options (Callable)
• Target Function
• Configuration
• Option metadata

Features

Configuration Editor

The configuration editor allows the user to specify a configuration-template using either (groups of) @dataclass-class or an ArgumentParser-instance. The editor will then automatically create a UI based on the provided template. Editors are specifically created for each option-property based on provided template-types (and extra constraints). Help-messages are displayed on hover, required arguments are highlighted when not filled in, etc.

We can also define our own option-source-method to dynamically create new option-groups based on the current configuration. This can be useful if we want to group options together, and only show certain groups when an attribute of another group is set to a certain value. E.g: only show ExtendedExampleModel-options if property model_type in MainOptions is set to "ExtendedExampleModel".

Configurations can be saved and loaded, a file-explorer view for the current workspace is made available.:

Run Queue

The run-queue window manages the currently running items. This could either be locally when using a local-app, or remotely, when using a client-app and a server-instance on which the actual Run-Queue is running. The Run-Queue allows us to add/remove items, pause/resume items, change the queue-order of items, and start autoprocessing, which will automatically start the next item in the queue when the current item is finished. We can set the number of processors as well, to run multiple items in parallel.

Configurations are passed to the user-provided run-function in separate processes. The stdout/stderr of each of the items is captured and displayed as a selectable console-output-view in the command-line-output window:

Remote-processing

Instead of using the local-app to manage and run the configurations on your own machine, we can use the client-app to connect to a server-instance on a remote machine. The client-app works analogous to the local-app and allows us to create and manage new configuration, but the Run-Queue runs on the connected remote Configurun-server:

Installation

This package can downloaded from this repository, or can be installed directly from PyPi by using pip:

pip install configurun

How to run?

Creating the app is done via the configurun.create-module. We can create 3 different types of apps:

• Local app - For running everything locally on your machine
• Client app - For running the configurations on a remote machine, connects to a server-instance
• Server instance - Command-line instance that listens to connections from client-apps. If login is succesful, accepts control of the RunQueue from the client-app.

On the client-side, the options_source should be set - the template of the settings used to create the configuration-editor.
On the server/running-machine, the target_function should be set - the function that actually runs the task/experiment (example).

Local App

A local app is an all-in-one app that can be used to create and run configurations locally on your machine. To run the example app, we can either call run_example_app() from configurun.examples or run the following code to construct the app ourselves:

### This example will run the app with an example configuration
# Also see `configurun/examples/example_target_function.py`
# Also see `configurun/examples/example_deduce_new_option_class_types.py`
import os
from configurun.create import local_app
from configurun.examples import example_target_function, example_deduce_new_option_classes

if __name__ == "__main__": #Makes sure bootstrapping process is done when running app
	local_app( #Create and runs a local configurun app-instance
		target_function=example_target_function, #The function that will be called with the configuration
		options_source=example_deduce_new_option_classes, #Template for UI-optiosn: Callable/@datclass/ArgumentParser
		workspace_path = os.path.join( #Settings, configs and the Run-Queue will be saved/loaded from/to here
			os.getcwd(), 
			"LocalExampleWorkspace"
		) 
	)

In this example, example_target_function runs a dummy task that logs to a file for 20 seconds. We can specify our own run-function to run our own scripts.

We can specify our own options source to create our own options-class for the configuration-editor, for example by using an existing ArgumentParser-object.

Client App

We can create a client-app and use it to login to running server-instances. We can then use the client-app analogous to the local-app to create new confiugrations and add/run/manage configurations on the remote machine.

# Opens a client-side app that we can use to connect to and control
# the server-instance
import os
from configurun.create import client
from configurun.examples import example_deduce_new_option_classes

if __name__ == "__main__":
	client(
		options_source=example_deduce_new_option_classes,
		workspace_path=os.path.join(os.getcwd(), "ClientExampleWorkspace"),
	)

Server-instance

The server-instance is a command-line app that listens to connections from client-instance(s) to receive new configurations and commands to manage its RunQueue. The actual run-functions are ran on this machine.

NOTE: after authentication, pickle/dill is used to transmit data, which indirectly enables arbitrary code execution on the server-side if the password is known. Please run the server on trusted network environments only. Run at your own risk!

# Opens a server-instance which tries to connect with clients and allows
# them to add configurations to the queue to be run on this machine
import os
from configurun.create import server
from configurun.examples.example_target_function import example_target_function

if __name__ == "__main__":
	# WARNING:
	# THIS ALLOWS OTHER MACHINES THAT RESIDE ON THE SAME NETWORK
	# TO EXECUTE ARBITRARY CODE ON THIS MACHINE IF THEY KNOW THE 
	# PASSWORD. PLEASE RUN IN A TRUSTED NETWORK ENVIRONMENT ONLY
	# RUN AT YOUR OWN RISK!
	server(
		target_function=example_target_function,
		workspace_path=os.path.join(os.getcwd(), "ServerExampleWorkspace"),
		password="password", #Password to connect to the server, make sure to change this!
		port=5454 #Port to connect to the server, defaults to 5454
	)

Option-source

When creating an app using the create-module, we can define a custom source, using the options_source=..., so we can construct the UI using our own options. We can use the following types as an options-source:

• @dataclass-object
• ArgumentParser-object
• Callable

Custom Options (@dataclass)

NOTE: Using fields results in more control over the final UI, for a more thorough example, please see this section and/or the example implementations in configurun/examples/example_options/example_options.py.

NOTE: When implementing custom option-classes, don't forget to add the @dataclass-decorator, and always inherit from BaseOptions

import os
from dataclasses import dataclass
from configurun.configuration.base_options import BaseOptions
from configurun.create import local_app
from configurun.examples import example_target_function


@dataclass #Don't forget to add this(!) - otherwise the app will not recognize the fields
class MyCustomOptions(BaseOptions): #Always inherit from BaseOptions (required to run config)
	simple_int : int = 1
	# etc...

if __name__ == "__main__":
	local_app(
		target_function=example_target_function,
		options_source=MyCustomOptions, #Simple: each configuration consists of a single options-class
		workspace_path = os.path.join(os.getcwd(), "ExampleDataclassOptions")
	)

Custom Options (ArgumentParser)

We can use a ArgumentParser-object as an options source, this will internally convert the argument parser into a @dataclass-object, which is then used as an options-class. Certain arguments are also parsed to control the UI (e.g. required=True, help="Will be displayed on hover").

import argparse
import os
from configurun.create import local_app
from configurun.examples import example_target_function

parser = argparse.ArgumentParser()
parser.add_argument("--required_arg", type=str, required=True, help="Required argument help")
#... add more arguments here

if __name__ == "__main__":
	local_app(
		target_function=example_target_function,
		options_source=parser, #Parser is converted internally to a dataclass-class which is used as the options-class
		workspace_path = os.path.join(os.getcwd(), "ExampleArgparseOptions")
	)

Custom Options (Callable)

A configuration is a collection of option-instances, which are grouped toghether in a Configuration-wrapper, which enables us to access the attributes of all enclosed options-instances using the configuration[attribute]/configuration.<attribute>/option_class.get(attribute, default). For more information, see this section.

We define an option-class as a class that has the @decorator and inherits from the BaseOptions-class.

As an options-source, we can create a callable which takes the current Configuration-instance as an argument and returns 1 or more new options-classes (not instances) which is called every time a setting is changed. If the types-change, the UI will be updated to reflect the new templates. This can be useful if we want to group options together, and only show certain groups when an attribute of another group is set to a certain value. For example:

#In this example, we will create a callable which returns new options-classes based on the 
# current configuration
import os
import typing
from dataclasses import dataclass
from configurun.create import local_app
from configurun.examples import example_target_function
from configurun.configuration import BaseOptions, Configuration

@dataclass #NOTE: Always use @dataclass for options
class AlwaysTheSame(BaseOptions): #NOTE: Always use BaseOptions as base class for options
	base_int : int = 1
	#...

@dataclass
class CustomOptionsDefault(BaseOptions):
	simple_int : int = 1
	#...

@dataclass
class CustomOptionsUnderConditions(BaseOptions):
	simple_int : int = 2
	some_more_options : str = 'Some string'
	#...

def deduce_new_option_classes(configuration: Configuration)\
		-> typing.Dict[str, typing.Type[BaseOptions | None]]: #Always return a dict of option 
	 		# classes the key of the dict is the name/group of the option-class
			# the value is the option-class (@dataclass & BaseOptions) itself 
	if configuration.options is None or len(configuration.options) == 0:
		pass #If initial configuration is being retrieved -> return default dict
	elif configuration.base_int == 2 and configuration.simple_int != 1:
		#Only return the CustomOptionsUnderConditions-class when base_int == 2 & simple_int != 1
		#NOTE: if we're not sure if attributes exist, we can use the `.get(key, default)` method
		return { #UI will be built using the following option-classes, each key gets a tab/window:
			'always_the_same' : AlwaysTheSame,
			'custom_options' : CustomOptionsUnderConditions
		}
	
	return { #UI will be built using the following option-classes, each key gets a tab/window:
		'always_the_same' : AlwaysTheSame,
		'custom_options' : CustomOptionsDefault
	} #NOTE: we must ALWAYS return a dictionary with at least 1 option class

if __name__ == '__main__':
	local_app(
		target_function=example_target_function,
		options_source=deduce_new_option_classes,
		workspace_path = os.path.join(os.getcwd(), "ExampleCallableOptions")
	)

Target Function

The target function is the function that does all the work. This is the function that is being called when an item starts "running" in the Run-Queue. It takes a single argument: a Configuration-instance - instance. The configuration-object contains all settings as set by the user when "add to queue" was pressed.

This example uses the example-configuration from the Configurtion section, we simply print the values of the configuration to the console:

def target_function(configuration: Configuration):
	#Do something with the configuration
	print(configuration.simple_int)
	print(configuration.some_other_int)
	#etc.

If you have replaced an argparse.Argumentparser in the previous example, this is the place where you insert the user-provided settings to the script that uses the ArgumentParser-object. For example:

# parsed_args = parser.parse_args() #will be done by user in UI
# your_framework_that_used_parsed_args(parsed_args) #Will be called in target_function

def target_function(configurtion : Configuration):
	# Since we can use the Configuration-instance as a dict 
	# and as configuration.<attribute> we can just
	# pass it to the framework compatible with the ArgumentParser:
	your_framework_that_used_parsed_args(configuration)

Configuration

Configurun works with configuration-objects. A configuration is a collection of option-instances (=@dataclass-instances that inherit from BaseOptions), which are grouped toghether in a Configuration-wrapper. We can think of the option-instances as the different groups of options we want to edit and use in our run (e.g. GeneralOptions(), LogOptions(), ModelOptions(), etc.). In the simplest case, we have 1 single option-instance which contains all the options, AllOptions().

The Configuration-wrapper enables us to access the attributes of all enclosed options-instances using configuration[attribute]/configuration.<attribute>/option_class.get(attribute, default).

An example of how to use a Configuration-instance:

from dataclasses import dataclass
from configurun.configuration import Configuration
from configurun.configuration import BaseOptions

@dataclass
class GeneralOptionsClass(BaseOptions):
	simple_int : int = 1
	#etc.

@dataclass
class OtherOptionClass(BaseOptions):
	some_other_int : int = 2
	#etc.


config = Configuration()
config.options['general_options'] = GeneralOptionsClass()
config.options['other_options'] = OtherOptionClass()

#Accessing the options, all of the following are equivalent:
print(config['simple_int'])
print(config.simple_int)
print(config.get('simple_int', -1)))

#These are also equivalent:
print(config['some_other_int'])
print(config.some_other_int)
print(config.get('some_other_int', -1)))

# Note that we can use the config.<attr>-notation to our advantage
# when we want to use autocomplete in our editor. For example:
# def target_function(configuration: GeneralOptionsClass)
# Would result in our editor of choice recognizing/autocompleting
# the `configuration.simple_hint` way of accessing `simple_hint`

Option metadata

The UI is mainly built around the field() functionality of python-dataclass, which allows the display-model to make use of the default values, type hints and other information. For each attribute in our option-definition, we can provide additional information in the metadata attribute of field(). This provides additional information to the UI, which is used to determine the editor-type, constraints etc.

For example:

from configurun.configuration import base_options
from dataclasses import field, dataclass
#Used to constrain the editors: (can also be imported from sklearn)
from pyside6_utils.utility.constraints import Interval 

@dataclass
class TestOptions(BaseOptions):
	test_int_property : int | None = field(
		default=None, #The default value used in the UI
		metadata=dict( #Contains additional information for the UI
			display_name="Test property", #The display-name
			help="This is a test property that can also be none", #On-hover help-messagem
			required=True, #If required, the field is red if not filled in
			constraints = [ #Limit editors (min/max, options, etc.)
				#The following constrains the editor to have value > 1
				Interval(type=int, left=1, right=None, closed="both"), 
				None #Or value can be None
			] 
			# etc...
		)
)

For more examples, please see the example-options.

The following metadata-keys are supported:

Metadata Key	Type	Description
"display_name"	str	Name to display for this attribute in the view - defaults to the variable name itself
"display_path"	str	Path to display this attribute - we can group/structure items. If parent does not exist, creates folders. Format as "
"help"	str	Help-message which will be shown when the user hovers over this item - empty by default
"constraints"	List[constraint]	Additional constraints on which the editor will be determined to apply to the field**, if none provided, use typehint of the field
"required"	bool	Whether this field is required to be filled in - if true - a red background will appear if the value is not set
"editable"	bool	Whether this field is editable - if false - the editor will be disabled

**: Constraints are sourced from the sklearn.utils._validation module and provides a way to constrain the dataclass fields such that the user can only enter valid values. They are also packed into the pyside6-utils package under utility.constraints. The following constraints are supported:

Constraint	Description	Editor Type
type	The type of the value should match the type of the constraint	based on type
Options / Container	The value should be one of the options provided in the constraint	QComboBox
StrOptions	The value should be one of the str-options provided in the constraint	QComboBox
Interval	The value should be within the interval provided in the constraint	QSpinBox or QDoubleSpinBox (limited)
None	None is a valid value for this field (same as typing.Optional)	Adds reset-button to editor
Range	The value should be within the range provided in the constraint	QSpinBox (limited)