Skip to main content

Programming- and CLI-Interface for the h5-dataformat of the Shepherd-Testbed

Project description

Core Library

PyPiVersion image Pytest CodeStyle

Documentation: https://orgua.github.io/shepherd/external/shepherd_core.html

Source Code: https://github.com/orgua/shepherd-datalib

Main Project: https://github.com/orgua/shepherd


This Python Module is designed as a library and bundles data-models and file-access-routines for the shepherd-testbed, that are used by several codebases.

For postprocessing shepherds .h5-files users want to use shepherd_data.

Features

  • read and write shepherds hdf5-files
  • create, read, write and convert experiments for the testbed (all data-models included)
  • simulate the virtual source, including virtual harvesters (and virtual converter as a whole)
  • connect and query the testbed via a webclient (TestbedClient)
    • offline usage defaults to static demo-fixtures loaded from yaml-files in the model-directories
  • work with target-firmwares
    • embed, modify, verify, convert
    • Note: working with ELF-files requires external dependencies, see Installation-Chapter
  • decode waveforms (gpio-state & timestamp) to UART
  • create an inventory (used versions of software, hardware)

See examples for more details and usage. Most functionality is showcased there. The extra-directory holds data-generators relevant for the testbed. Notably is a trafficbench-experiment that's used to derive the link-matrix.

Compatibility

OS PyVersion Comment
Ubuntu 3.8 - 3.12
Windows 3.8 - 3.12 no support for elf and hex-conversions yet
MacOS 3.8 - 3.12 hex-conversion missing

Notes:

  • hex-conversion needs a working and accessible objcopy
  • elf-supports needs
    • shepherd-core[elf] installs pwntools-elf-only
    • most elf-features also still utilize hex-conversion

Data-Models in Detail

  • new orchestration /data-models with focus on remote shepherd-testbed
  • classes of sub-models
    • /base: base-classes, configuration and -functionality for all models
    • /testbed: meta-data representation of all testbed-components
    • /content: reusable meta-data for fw, h5 and vsrc-definitions
    • /experiment: configuration-models including sub-systems
    • /task: digestible configs for shepherd-herd or -sheep
    • behavior controlled by ShpModel and content-model
  • a basic database is available as fixtures through a tb_client
    • fixtures selectable by name & ID
    • fixtures support inheritance
  • models support
    • auto-completion with neutral / sensible values
    • complex and custom datatypes (i.e. PositiveInt, lists-checks on length)
    • checking of inputs and type-casting
    • generate their own schema (for web-forms)
    • pre-validation
    • store to & load from yaml with typecheck through wrapper
    • documentation
  • experiment-definition is designed securely
    • types are limited in size (str)
    • exposes no internal paths
  • experiments can be transformed to task-sets (TestbedTasks.from_xp())

Installation

The Library is available via PyPI and can be installed with

  pip install shepherd-core -U

  # or for the full experience (includes core)
  pip install shepherd-data -U

For bleeding-edge-features or dev-work it is possible to install directly from GitHub-Sources (here dev-branch):

pip install git+https://github.com/orgua/shepherd-datalib.git@dev#subdirectory=shepherd_core -U

If you are working with .elf-files (embedding into experiments) you make "objcopy" accessible to python. In Ubuntu, you can either install build-essential or binutils-$ARCH with arch being msp430 or arm-none-eabi for the nRF52.

  sudo apt install build-essential

For more advanced work with .elf-files (modify value of symbols / target-ID) you should install

  pip install shepherd-core[elf]

and also make sure the prereqs for the pwntools are met.

For creating an inventory of the host-system you should install

  pip install shepherd-core[inventory]

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

shepherd_core-2023.11.1.tar.gz (90.7 kB view hashes)

Uploaded Source

Built Distribution

shepherd_core-2023.11.1-py3-none-any.whl (129.8 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page