Python scripts for operation of FAME models
Project description
FAME-Io
Python scripts for FAME models, generation of protobuf input files and conversion of protobuf output files. Please visit the FAME-Wiki to get an explanation of FAME and its components. The package is also formerly known as FAME-Py.
Installation
pip install fameio
You may also use pipx
. For detailed information please refer to the official pipx
documentation.
pipx install fameio
Usage
FAME-Io currently offers two main scripts "makeFameRunConfig" and "convertFameResults". Both are automatically installed with the package. The first one creates a protobuf file for FAME applications using YAML definition files and .csv files. The latter one reads output files from FAME applications in protobuf format and converts them to .csv files.
Make a FAME run configuration
Digests configuration files in YAML format, combines them with .csv data files and creates a single input file for FAME applications in protobuf format. Call structure:
makeFameRunConfig -f <path/to/scenario.yaml>
You may also specify any of the following arguments:
Command | Action |
---|---|
-l or --log |
Sets the logging level. Default is info . Options are debug , info , warning , warn , error , critical . |
-lf or --logfile |
Sets the logging file. Default is None . If None is provided, all logs get only printed to the console. |
-o or --output |
Sets the path of the compiled protobuf output file. Default is config.pb . |
This could look as follows:
makeFameRunConfig -f <path/to/scenario.yaml> -l debug -lf <path/to/scenario.log> -o <path/to/config.pb>
You may also call the configuration builder from any Python script with
from fameio.scripts.make_config import run as make_config
make_config("path/to/scenario.yaml")
Similar to the console call you may also specify custom run config arguments and add it in a dictionary to the function call.
from fameio.scripts.make_config import run as make_config
from fameio.source.cli import Options
run_config = {Options.LOG_LEVEL: "info",
Options.OUTPUT: "output.pb",
Options.LOG_FILE: "scenario.log",
}
make_config("path/to/scenario.yaml", run_config)
Scenario YAML
The "scenario.yaml" file contains all configuration options for a FAME-based simulation.
It consists of the sections Schema
, GeneralProperties
, Agents
and Contracts
, all of them described below.
Schema
The Schema is used to validate the inputs of the "scenario.yaml". Since the Schema should be valid across multiple scenarios, it is recommended to defined it in a separate file and include the file here.
Currently, the schema specifies:
- which type of Agents can be created
- what type of input attributes an Agent uses
- what type of Products an Agent can send in Contracts.
The Schema consists of the sections Header
and AgentTypes
.
Header
Scientific applications often evolve, and so do their required input parameters. Therefore, the header specifies information what FAME-based application the schema is corresponding to. In this way a schema.yaml is tied to a specific version an application, ensuring a match between the inputs required by the application, and those provided by the files created with FAME-Io.
Header:
Project: MyProjectName
RepoUrl: https://mygithosting.com/myProject
CommitHash: abc123
Project
name of your project / FAME-based applicationRepoUrl
URL of your projectCommitHash
hash of the commit / version of your project
AgentTypes
Here, each type of agent that can be created in your FAME-based application is listed, its attributes and its available Products for Contracts. The structure of this section
AgentTypes:
MyAgentType:
Attributes:
MyAttribute:
...
MyOtherAttribute:
...
Products: [ 'Product1', 'Product2', 'Product3' ]
MyOtherAgentWithoutProductsOrAttributes:
MyAgentType
Java's simple class name of the Agent typeAttributes
indicates that beginning of the attribute definition section for this Agent typeMyAttribute
Name of an attribute as specified in the corresponding Java source code of this Agent type (annotated with "@Input")MyOtherAttribute
Name of another attribute derived from Java source codeProducts
list of Products that this Agent can send in Contracts; derived from Java source code of this Agent type (annotated with "@Product")MyOtherAgentWithoutProductsOrAttributes
an Agent type that requires neither Attributes nor Products
Both Attributes and Products are optional - there could be useful Agents that require neither of them.
In the above example attribute definition was not shown (indicated by ...
).
The next example provides details on how to define an attribute:
MySimpleAttribute:
AttributeType: enum
Mandatory: true
List: false
Values: [ 'AllowedValue1', 'AllowedValue2' ]
Default: 'AllowedValue1'
Help: 'My help text'
MyComplexAttribute:
AttributeType: block
NestedAttributes:
InnerAttributeA:
AttributeType: integer
InnerAttributeB:
AttributeType: double
MySimpleAttribute
,MyDoubleList
,MyComplexAttribute
Names of the attributes as specified in the Java enum annotated with "@Input"AttributeType
(required) data type of the attribute; see options in table belowMandatory
(optional - true by default) if true: the attribute is required for this agent and validation will fail if the attribute is missing in the scenario and no default is providedList
(optional - false by default)AttributeType: time_series
cannot be trueAttributeType: block
- if true: any nested element in the scenario must be part of a list element and thus can appear multiple times
- if false: any nested element in the scenario can only appear once
- any other AttributeType: the attribute is interpreted as list, i.e. multiple values can be assigned to this attribute in the scenario
NestedAttributes
(required only ifAttributeType: block
, otherwise disallowed) starts an inner Attribute definition block - defined Attributes are sub-elements ofMyComplexAttribute
Values
(optional - None by default): if present defines a list of allowed values for this attributeDefault
(optional - None by default): if present defines a default value to be used in case the scenario does not specify itHelp
(optional - None by default): if present defines a help text to you attribute
AttributeType | value |
---|---|
integer |
a 32-bit integer value |
double |
a 64-bit floating-point value (integers also allowed) |
long |
a 64-bit integer value |
time_stamp |
either a FAME time stamp string or 64-bit integer value |
string |
any string |
enum |
any string, however, usually tied to a set of allowed Values |
time_series |
either a path to a .csv-file or a single 64-bit floating-point value; does not support List: true |
block |
this attribute has no value of its own but hosts a group of nested Attributes; implies NestedAttributes to be defined |
GeneralProperties
Specifies FAME-specific properties of the simulation. Structure:
GeneralProperties:
RunId: 1
Simulation:
StartTime: 2011-12-31_23:58:00
StopTime: 2012-12-30_23:58:00
RandomSeed: 1
Output:
Interval: 100
Process: 0
Parameters:
RunId
an ID that can be given to the simulation; use at your discretionStartTime
time stamp in the format YYYY-MM-DD_hh:mm:ss; first moment of the simulation.StopTime
time stamp in the format YYYY-MM-DD_hh:mm:ss; last moment of the simulation - i.e. simulation terminates after passing that time stampRandomSeed
seed to initialise random number generation; each value leads to a unique series of random numbers.Interval
number of simulation ticks in between write-to-disk events; may be used for performance optimisations;Process
id of process that performs write-to-disk operations; leave at 0 to be compatible with single-processes;
Agents
Specifies all Agents to be created in the simulation in a list. Each Agent has its own entry. Structure:
Agents:
- Type: MyAgentWithInputs
Id: 1
Attributes:
MyEnum: SAME_SHARES
MyInteger: 2
MyDouble: 4.2
MyTimeSeries: "./path/to/time_series.csv"
- Type: MyAgentWithoutInputs
Id: 2
Agent Parameters:
Type
Mandatory; Java's simple class name of the agent to be createdId
Mandatory; simulation-unique id of this agent; if two agents have the same ID, the configuration process will stop.Attributes
Optional; if the agent has any attributes, specify them here in the format "AttributeName: value"; please see attribute table above
The specified Attributes
for each agent must match the specified Attributes
options in the linked Schema (see above).
For better structure and readability of the scenario.yaml
, Attributes
may also be specified in a nested way as demonstrated below.
Agents:
- Type: MyAgentWithInputs
Id: 1
Attributes:
Parent:
MyEnum: SAME_SHARES
MyInteger: 2
Parent2:
MyDouble: 4.2
Child:
MyTimeSeries: "./path/to/time_series.csv"
In case Attributes are defined with List: true
option, lists are assigned to an Attribute or Group:
Attributes:
MyDoubleList: [5.2, 4.5, 7, 9.9]
MyListGroup:
- IntValueA: 5
IntValueB: 42
- IntValueA: 7
IntValueB: 100
Here, MyDoubleList
and MyListGroup
need to specify List: true
in the corresponding Schema.
The shorter []
-notation was used to assign a list of floating-point values to MyDoubleList
.
Nested items IntValueA
and IntValueB
of MyListGroup
are assigned within a list, allowing the specification of these nested items several times.
Contracts
Specifies all Contracts, i.e. repetitive bilateral transactions in between agents.
Contracts are given as a list.
We recommend moving Contracts to separate files and to use the !include
command to integrate them in the scenario.
Contracts:
- SenderId: 1
ReceiverId: 2
ProductName: ProductOfAgent_1
FirstDeliveryTime: -25
DeliveryIntervalInSteps: 3600
- SenderId: 2
ReceiverId: 1
ProductName: ProductOfAgent_2
FirstDeliveryTime: -22
DeliveryIntervalInSteps: 3600
Attributes:
ProductAppendix: value
TimeOffset: 42
Contract Parameters:
SenderId
unique ID of agent sending the productReceiverId
unique ID of agent receiving the productProductName
name of the product to be sentFirstDeliveryTime
first time of delivery in the format "seconds after the January 1st 2000, 00:00:00"DeliveryIntervalInSteps
delay time in between deliveries in secondsAttributes
can be set to include additional information asint
,float
,enum
ordict
data types
Definition of Multiple Similar Contracts
Often, scenarios contain multiple agents of similar type that also have similar chains of contracts.
Therefore, FAME-Io supports a compact definition of multiple similar contracts.
SenderId
and ReceiverId
can both be lists and support One-to-N, N-to-One and N-to-N relations like in the following example:
Contracts:
# effectively 3 similar contracts (0 -> 11), (0 -> 12), (0 -> 13)
# with otherwise identical ProductName, FirstDeliveryTime & DeliveryIntervalInSteps
- SenderId: 0
ReceiverId: [11, 12, 13]
ProductName: MyOtherProduct
FirstDeliveryTime: 100
DeliveryIntervalInSteps: 3600
# effectively 3 similar contracts (1 -> 10), (2 -> 10), (3 -> 10)
# with otherwise identical ProductName, FirstDeliveryTime & DeliveryIntervalInSteps
- SenderId: [1, 2, 3]
ReceiverId: 10
ProductName: MyProduct
FirstDeliveryTime: 100
DeliveryIntervalInSteps: 3600
# effectively 3 similar contracts (1 -> 11), (2 -> 12), (3 -> 13)
# with otherwise identical ProductName, FirstDeliveryTime & DeliveryIntervalInSteps
- SenderId: [1, 2, 3]
ReceiverId: [11, 12, 13]
ProductName: MyThirdProduct
FirstDeliveryTime: 100
DeliveryIntervalInSteps: 3600
Combined with YAML anchors complex contract chains can be easily reduced to a minimum of required configuration. The following example is equivalent to the previous one and allows a quick extension of contracts to a new couple of agents e.g. (4;14):
Groups:
- &agentList1: [1,2,3]
- &agentList2: [11,12,13]
Contracts:
- SenderId: 0
ReceiverId: *agentList2
ProductName: MyOtherProduct
FirstDeliveryTime: 100
DeliveryIntervalInSteps: 3600
- SenderId: *agentList1
ReceiverId: 10
ProductName: MyProduct
FirstDeliveryTime: 100
DeliveryIntervalInSteps: 3600
- SenderId: *agentList1
ReceiverId: *agentList2
ProductName: MyThirdProduct
FirstDeliveryTime: 100
DeliveryIntervalInSteps: 3600
CSV files
TIME_SERIES inputs are not directly fed into the Scenario YAML file. Instead, TIME_SERIES reference a .csv file that can be stored some place else. These .csv files follow a specific structure:
- They must contain exactly two columns.
- The first column must be a time stamp in form
YYYY-MM-DD_hh:mm:ss
- The second column must be a numerical value (either integer or floating-point)
- The separator of the two columns is a semicolon
Exemplary content of a valid .csv file:
2012-01-01_00:00:00;400
2013-01-01_00:00:00;720.5
2014-01-01_00:00:00;650
2015-01-01_00:00:00;99.27772
2016-01-01_00:00:00;42
2017-01-01_00:00:00;0.1
Please refer also to the detailed article about TimeStamps
in the FAME-Wiki.
Split and join multiple YAML files
The user may include other YAML files into a YAML file to divide the content across files as convenient.
We explicitly recommend using this feature for the Schema
and Contracts
sections.
Otherwise, the scenario.yaml may become crowded.
Command: !Include
To hint YAML to load the content of another file use !include "path/relative/to/including/yaml/file.yml"
.
You can concatenate !include commands and can use !include in the included file as well.
The path to the included file is always relative to the file using the !include command.
So with the following file structure
file-structure
a.yaml
folder/b.yaml
folder/c.yaml
folder/deeper_folder/d.yaml
the following !include commands work
in a.yaml
ToBe: !include "folder/b.yaml"
OrNot: !include "folder/deeper_folder/d.yaml"
in b.yaml
ThatIs: !include "c.yaml"
TheQuestion: !include "deeper_folder/d.yaml"
Provided that
in c.yaml
Or: maybe
d.yaml
not: "?"
the resulting file would look like this:
THe Joined file a.yaml
ToBe:
ThatIs:
Or: maybe
TheQuestion:
not: "?"
OrNot:
not: "?"
You may also specify absolute file paths if preferred by starting with a "/".
When specifying only a file path, the complete content of the file is assigned to the given key. You always need a key to assign the !include command to. However, you cannot combine the value returned from !include with other values in the same key. Thus, the following combinations do not work:
caveats.yml
!include "file.yaml" # no key assigned
Key:
Some: OtherItem
!include "file.yaml" # cannot join with other named items
List:
- an: entry
!include "file.yaml" # cannot directly join with list items, even if !include returns a list
Integrate specific nodes of YAML files
Instead of including all content in the included file, you may also pick a specific node within that file.
For this use !include [<relative/path/to/file.yaml>, Path:To:Field:In:Yaml]
.
Here, :
is used in the node-specifying string to select a sequence of nodes to follow - with custom depth.
Consider the following two files:
file_to_be_included.yaml
Set1:
Subset1:
Key: Value
Set2:
OtherKey: OtherValue
including_file.yaml
- Type: MyAgentWithInputs
Id: 1
Attributes: !include_node [file_to_be_included.yaml, Set1:Subset1]
Compiling "including_file.yaml" results in
resulting_file.yaml
- Type: MyAgentWithInputs
Id: 1
Attributes:
Key: Value
Load multiple files
Using wildcards in the given path (e.g. "path/to/many/*.yaml") will lead to loading multiple files and assigning their content to the same key. You can make use of this feature with or without specifying a node selector. However, the elements to be joined across multiple files must be lists. These lists are then concatenated into a single list and then assigned to the key in the file calling !include. This feature is especially useful for Contracts: You can split the Contracts list into several files and place them in a separate folder. Then use !include to re-integrate them into your configuration. An example:
my_contract1.yaml
Contracts:
- ContractA
- ContractB
my_contract2.yaml
Contracts:
- ContractC
- ContractD
- ContractE
including_file.yaml
Contracts: [!include "my_contract*.yaml", "Contracts"]
results in
result.yaml
Contracts:
- ContractA
- ContractB
- ContractC
- ContractD
- ContractE
Ignoring files
Files that have their name start with "IGNORE_" are not included with the !include command. You will see a debug output to notify you that the file was ignored. Use this to temporarily take files out ouf your configuration without deleting or moving them.
Read FAME results
Takes an output file in protobuf format of FAME-based applications and converts it into files in .csv format. An individual file for each type of Agent is created in a folder named after the protobuf input file. Call structure:
convertFameResults -f <./path/to/protobuf_file.pb>
You may also specify any of the following arguments:
Command | Action |
---|---|
-l or --log |
Sets the logging level. Default is info . Options are debug , info , warning , warn , error , critical . |
-lf or --logfile |
Sets the logging file. Default is None . If None is provided, all logs get only printed to the console. |
-a or --agents |
If specified, only a subset of agents is extracted from the protobuf file. Default is to extract all agents. |
-o or --output |
Sets the path to where the generated output files are written to. If not specified, the folder's name is derived from the input file's name. Folder will be created if it does not exist. |
-se or --single-export |
Enables export of individual agents to individual files, when present. If not present (the default) one file per AgentType is created. |
-m or --memory-saving |
When specified, reduces memory usage profile at the cost of runtime. Use only when necessary. |
-cc or --complex-column |
Defines how to deal with complex indexed output columns (if any). IGNORE ignores complex columns. MERGE squashes all data from complex columns in one big string entry. SPLIT creates a separate file for each complex indexed output column. |
This could look as follows:
convertFameResults -f <./path/to/protobuf_file.pb> -l debug -lf <path/to/output.log> -a AgentType1 AgentType2 -o myCsvFolder -m -cc SPLIT
You may also call the conversion script from any Python script with
from fameio.scripts.convert_results import run as convert_results
convert_results("./path/to/protobuf_file.pb")
Similar to the console call you may also specify custom run config arguments and add it in a dictionary to the function call.
from fameio.scripts.convert_results import run as convert_results
from fameio.source.cli import Options
run_config = {Options.LOG_LEVEL: "info",
Options.LOG_FILE: "scenario.log",
Options.OUTPUT: "Output",
Options.AGENT_LIST: ['AgentType1', 'AgentType2'],
Options.MEMORY_SAVING: False,
Options.SINGLE_AGENT_EXPORT: False,
Options.RESOLVE_COMPLEX_FIELD: "SPLIT",
}
convert_results("./path/to/protobuf_file.pb", run_config)
Contribute
Please read the Contributors License Agreement cla.md
, sign it and send it to fame@dlr.de
before contributing.
Testing changes locally
Once some changes have been performed on the local git clone, use the following command to override your local installation with your modified copy in order to test the result:
python3 setup.py bdist_wheel && pip3 install --force-reinstall --no-dependencies ./dist/*.whl
Code style
We use the code formatting library black
.
The maximum line length is defined as 120 characters.
Therefore, before committing, run black --line-length 120 .
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.