Skip to main content

Programming- and CLI-Interface for the h5-dataformat of the Shepherd-Testbed

Project description

Core Library

PyPiVersion image Pytest CodeStyle

Main Documentation: https://orgua.github.io/shepherd

Source Code: https://github.com/orgua/shepherd-datalib

Main Project: https://github.com/orgua/shepherd


shepherd-core is designed as a library and bundles data-models and file-access-routines for the shepherd-testbed, that are used by several codebases.

For postprocessing shepherds .h5-files usage of shepherd_data is recommended.

Features

  • read and write shepherds hdf5-files
  • create, read, write and convert experiments for the testbed
    • all required data-models are included
  • simulate the virtual source, including virtual harvesters (and virtual converter as a whole)
  • connect and query the testbed via a webclient (TestbedClient in alpha-stage)
    • offline usage defaults to static demo-fixtures loaded from yaml-files in the model-directories
  • work with target-firmwares
    • embed, modify, verify, convert
    • Note: working with ELF-files requires external dependencies, see Installation-Chapter
  • decode waveforms (gpio-state & timestamp) to UART
  • create an inventory (for deployed versions of software, hardware)

See official documentation or example scripts for more details and usage. Most functionality is showcased in both. The extra-directory holds data-generators relevant for the testbed. Notably is a trafficbench-experiment that's used to derive the link-matrix of the testbed-nodes.

Config-Models in Detail

These pydantic data-models are used throughout all shepherd interfaces. Users can create an experiment, include their own content and feed it to the testbed.

  • orchestration /data-models with focus on remote shepherd-testbed
  • classes of sub-models
    • /base: base-classes, configuration and -functionality for all models
    • /testbed: meta-data representation of all testbed-components
    • /content: reusable user-defined meta-data for fw, h5 and vsrc-definitions
    • /experiment: configuration-models including sub-systems
    • /task: digestible configs for shepherd-herd or -sheep
    • behavior controlled by ShpModel and content-model
  • a basic database is available as fixtures through a tb_client
    • fixtures selectable by name & ID
    • fixtures support inheritance
  • the models support
    • auto-completion with neutral / sensible values
    • complex and custom datatypes (i.e. PositiveInt, lists-checks on length)
    • checking of inputs and type-casting
    • generate their own schema (for web-forms)
    • pre-validation
    • store to & load from yaml with typecheck through wrapper
    • documentation
  • experiment-definition is designed securely
    • types are limited in size (str)
    • exposes no internal paths
  • experiments can be transformed to task-sets (TestbedTasks.from_xp())

Compatibility

OS PyVersion Comment
Ubuntu 3.8 - 3.13
Windows 3.8 - 3.13 no support for elf and hex-conversions yet
MacOS 3.8 - 3.13 hex-conversion missing

Notes:

  • hex-conversion needs a working and accessible objcopy
  • elf-supports needs
    • shepherd-core[elf] installs pwntools-elf-only
    • most elf-features also still utilize hex-conversion

Installation

The Library is available via PyPI and can be installed with

pip install shepherd-core -U

# or for the full experience (includes core)
pip install shepherd-data -U

For bleeding-edge-features or dev-work it is possible to install directly from GitHub-Sources (here dev-branch):

pip install git+https://github.com/orgua/shepherd-datalib.git@dev#subdirectory=shepherd_core -U
# and on sheep with newer debian
sudo pip install git+https://github.com/orgua/shepherd-datalib.git@dev#subdirectory=shepherd_core -U --break-system-packages

If you are working with .elf-files (embedding into experiments) you make "objcopy" accessible to python. In Ubuntu, you can either install build-essential or binutils-$ARCH with arch being msp430 or arm-none-eabi for the nRF52.

sudo apt install build-essential

For more advanced work with .elf-files (modify value of symbols / target-ID) you should install

pip install shepherd-core[elf]

and also make sure the prereqs for the pwntools are met.

For creating an inventory of the host-system you should install

pip install shepherd-core[inventory]

Unittests

To run the testbench, follow these steps:

  1. Navigate your host-shell into the package-folder and
  2. install dependencies
  3. run the testbench (~ 320 tests):
cd shepherd-datalib/shepherd_core
pip3 install ./[tests]
pytest

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

shepherd_core-2024.9.1.tar.gz (538.8 kB view hashes)

Uploaded Source

Built Distribution

shepherd_core-2024.9.1-py3-none-any.whl (113.8 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page