robotframework-construct 0.3.0

Infrastructure to use construct Structs in robotframework

robotframework-construct

I am in a hurry, let's jump-start with an example!

git (a version control system) and uv (a tool for managing Python virtual environments) need to be preinstalled to run the examples.

git clone https://github.com/MarketSquare/robotframework-construct.git
cd robotframework-construct
uv sync --extra test --dev
uv run xonsh tasks/baseQC.xsh

Some examples, such as USB HID and nfc/nci, require specific hardware to function. For the nci example, STm hardware and firmware is required, which can be requested from [e-mail]. For the HID example, a USB Keyboard on a linux machine is sufficient.

What is robotframework-construct?

robotframework-construct is a Robot Framework keyword library powered by construct.

construct is a declarative and symmetrical parser and builder for binary data.

Aiming for :rocket: speed, :white_check_mark: reliability, and :microscope: visibility.

Your binary data becomes as accessible as numbers and strings are in Robot Framework.

Check out the documentation here

Licensing

robotframework-construct is an opensource keyword library licensed under Apache-2.0, leveraging the construct library licensed under MIT.

Use cases

• Beautifully access registers, for both reading and writing.
• Test your production construct specification against a reference implementation of the same protocol.
• Test your production binary parser/generator against a construct implementation of your binary format.
• Use your construct declarations to:
  • Craft intentionally corrupted data.
  • Fuzz test your binary parsers.
• While the network is an interesting area for robotframework-construct, other interfaces (UART/SPI/I2C) are considered first-class citizen.

Features

Check out the full documentation at robotframework-construct.

From construct, declaration not implementation of a parser/generator

This is a real-world usable declaration of the bson protocol.

from construct import Struct, Int8ul, Int32sl, Int64sl, Int64ul, Float64l, Const, Array, Byte, GreedyBytes, CString, Prefixed, Switch, LazyBound, Pass, GreedyRange, Rebuild, this


_bson_element = Struct(
    "type" / Int8ul,
    "name" / CString("utf8"),
    "value" / Switch(this.type, {
         1: Float64l,
         2: Prefixed(Int32sl, CString("utf8")),
         3: LazyBound(lambda: document),
         4: LazyBound(lambda: document),
         5: Prefixed(Int32sl, GreedyBytes),
         6: Pass,
         7: Array(12, Byte),
         8: Int8ul,
         9: Int64sl,
        10: Pass,
        11: Struct("pattern" / CString("utf8"), "options" / CString("utf8")),
        12: Struct("namespace" / CString("utf8"), "id" / Array(12, Byte)),
        13: Prefixed(Int32sl, CString("utf8")),
        14: Prefixed(Int32sl, CString("utf8")),
        15: Struct("code" / Prefixed(Int32sl, CString("utf8")), "scope" / LazyBound(lambda: document)),
        16: Int32sl,
        17: Int64ul,
        18: Int64sl,
        19: Array(16, Byte),
        -1: Pass,
        127: Pass,
    })
)
_e_list = GreedyRange(_bson_element)

def _calc_size(this):
    return  len(_e_list.build(this["elements"]))+5

document = Struct(
document = Struct(
    "size" / Rebuild(Int32sl, _calc_size),
    "elements" / _e_list,
    "EOO" / Const(b"\x00")
)

This can be readily and directly derived from the bson specification. AI can assist in this process efficiently. This is because the mapping between the specification and the declaration is a very direct and straightforward task, making it easy to supervise the process and verify the result.

Using AI to generate a parser+generator would result in a larger volume of code to be verified, and the verification is harder.

Checking and modifying binary data

There are keywords with embedded parameters that allow checking and modifying binary data in a Robot Framework way

A checking example

and a modifying example

Note: This is very natural in the Robot Framework environment. If multiple elements need to be checked, these checks should be organized in keywords.

Observing the binary data

The built and parsed binary data is easily accessible. This helps with trust issues and makes it easier to read digital analyzer or oscilloscope screens. Also, a name to identify what definition is doing the parsing/generating may be provided.

A building example:

A parsing example:

Breaking out of the ecosystems

The highly valuable building/parsing infrastructure does not depend on robotframework, and in the case of the parsing part, it also does not depend on Python. The Structs can be transformed into kaitai. Kaitai is a DSL that can be transformed into parsers in 10+ languages and counting, you find further details here.

Keep in mind that some limitations apply to these transformations.

For reference: [./tasks/breakoutCpp.xsh], which is a script that demonstrates how to transform Construct declarations into a C++ parser using the Kaitai DSL.

Relationships in the Ecosystem

The number of dependencies is kept low, with no transitive dependencies.

This is important as it keeps coordination feasible. Construct is well-developed and not expected to change significantly soon. Robot Framework releases major updates annually, but these are well-managed and communicated.

Construct

All parsing and generating capabilities come from Construct. No additional parsing/generating code is added; the only code added interfaces Construct with Robot Framework. The way Construct objects are created remains unchanged.

Construct has no non-optional dependencies.

Robot Framework

This project connects Construct with Robot Framework. Only official APIs are used, and this project depends entirely on Robot Framework.

Robot Framework has no non-optional dependencies.

Rammbock

Rammbock inspired this project, as it was one of the reasons I started using Robot Framework.

Instead of maintaining Rammbock, we chose to integrate Construct.

Reasoning

Both Rammbock and Construct have limited engineering resources, but Construct is currently better supported. Construct also collaborates with Kaitai, engaging communities in C#, C++, and other ecosystems.

Using Construct provides a clear separation between parsing/generating logic and interface code, enabling expansion into other ecosystems.

Installation

The robotframework-construct keyword library is hosted on pypi and can be installed like any pypi hosted python dependency with pip.

pip install robotframework-construct

Limitations

Construct declarations must be written in .py files. There are no plans to integrate the Construct DSL into Robot Framework.

This eases the breaking out of the robot-framework and Python ecosystems.

Quality Control Measures

Tested examples and acceptance tests using Robot Framework are provided. Unit tests are not a priority.

Mutation Testing

Since this project primarily consists of interface code, it is crucial to catch user errors and produce clear error messages. Mutation testing ensures that all code paths and error messages are tested, supporting efforts to make errors informative.

Project To-Do List

• Parsing functionality demonstrated with an in-memory BSON object.
• Parsing functionality demonstrated with a BSON file.
• Generating functionality demonstrated with an in-memory BSON object.
• Generating functionality demonstrated with a binary file.
• Register read/write demonstrated with a mockup register.
• Receive/transmit network example using DNS.
• Reflector tool to allow to implement servers using clients.
• Upload wheel to pypi.
• Increase test coverage (Mutant killing) of the reflector
• Segmentise mutation testing to speedup
• Comment and document the real world example with the USB HID keyboard
• Add a second real world example with binary interface to Readme
• Have libdoc documentation online
• Have libdoc documentation online for all keywords, not only the central ones
• User guide and tutorials/Article for (https://medium.com/@RobotFramework/).
• Example on how to breakout of the python ecosystem
• Midway review with Robot Framework Foundation.
• Final review with Robot Framework Foundation.