ecidcodes 0.2.8

Python bindings for ecidcodes Library

IDCODES Library

Overview

IDCODES is a C++ library designed to provide tools for Binary Galois (Finite) Field GF and includes functionality for polynomial evaluation RSID and RS2ID, hashing SHA1ID and SHA256ID, Reed-Muller RMID, and collision testing modules. The library is designed for use in high-performance computational environments and supports modular, extensible templates.

Features

• Galois Field Operations: Initialization, multiplication, and evaluation of polynomials over finite fields GF 2^n = 1 to 64.
• Reed Solomon Codes: Evaluate Reed-Solomon and Concatenated Reed-Solomon Tagging codes
• Hashing Algorithms: Implements SHA-1 and SHA-256 for secure message hashing.
• Reed-Muller Codes: Evaluate multivariable polynomials for error correction.
• Collision Testing: Measure collision probabilities in tag values for large-scale simulations.
• Speed Benchmarks: Measure execution speed for various parameters with CLI input and save to CSV Results.

Requirements

• C++: Requires a modern C++ compiler with support for C++17 and CMake.

  On Ubuntu the C/C++ compiler and cmake can be installed with:

  sudo apt update
  sudo apt install build-essential cmake
  

• Dependencies:

  • Boost (for UUID and SHA-1)
  • OpenSSL (for cryptographic functions and SHA-256)
  • LibSSL API
  • Python 3.12

  On Ubuntu Boost, OpenSSL and Libssl can be installed with the following commands:

  sudo apt install libboost-all-dev openssl libssl-dev
  

Build Instructions

Follow these steps to install dependencies, compile and build the project:

1. Clone the repository and checkout development branch:

   # Over HTTPS (requires login with Github credentials)
   git clone https://github.com/Ecologic-Computing/idcodes.git
   # Over SSH (requires SSH key to be added to Github account)
   git clone [e-mail]:Ecologic-Computing/idcodes.git
   cd idcodes
   git checkout release
   

2. Install necessary dependencies on Ubuntu using install_dependencies.sh script:

   chmod +x install_dependencies.sh
   sudo ./install_dependencies.sh
   

3. Create a build directory in the repository root folder:

   mkdir build
   cd build
   

4. Run cmake to generate the build files & Compile the code using make:

   cmake ..
   make
   

5. Install Scripts for benchmarking and Other Python files:

   make install 
   

6. Create a Python 3.12 Virtual environment at location ./.venv using requirements.txt file to run python files:

   bash ./python_venv.sh
   

7. Activate python environment:

   source .venv/bin/activate
   

Using CMakePresets.json to build and install:

From the root directory of idcodes repo run the following commands to build and install See buildPresets in CMakePresets.json for common build types such as Release and Debug as well as Platform Architecture for Intel/AMD x86_64 or for ARM aarch64

1. To configure current Preset run:

   cmake --preset <buildPresets.name>
   

2. To build run:

   cmake --build --preset <buildPresets.name>
   

3. To install run:

   cmake --build --target install --preset <buildPresets.name>
   

Example

This example shows benchmark CLI utility:

$ source .venv/bin/activate
$ cd out/build/GCC_ARM
$ ./bin/benchmarks --help
Idcodes Library Benchmark 

Allowed options:
  --help                       help message
  --n_iter arg (=100000)       Number of iterations for benchmarking, positive 
                               integer >0
  --vec_len arg                Vector length in Bytes for benchmarking, positive integer
                               >0
  --id_code_type arg           ID code type: SHA256ID | SHA1ID | RSID | RS2ID |
                               RMID
  --tag_pos arg (=2)           Tag Position for encoding: Positive integer >=0
  --gf2_exp arg                GF(2^n) Exponent values,  positive integer >=0 
                               max 64
  --data_type arg (=txt)       Data Type: txt | bin | random
  --input_data arg             Message Input File, full path
  --rs2_inner_tag_pos arg (=2) Concatenated RS Inner tag position, positive 
                               integer >=0
  --rm_order arg (=)          Reed Muller monomial order, positive integer >=0
  --csv_file_path arg (=./)    CSV file path Benchmark Results, make sure Path 
                               exists on system!

Main Function:

$ ./main --help
Idcodes Library Main Function 

Allowed options:
  --help                       help message
  --vec_len arg                Vector length in Bytes for benchmarking, positive integer
                               >0
  --id_code_type arg           ID code type: SHA256ID | SHA1ID | RSID | RS2ID |
                               RMID
  --tag_pos arg (=2)           Tag Position for encoding: Positive integer >=0
  --gf2_exp arg                GF(2^n) Exponent values,  positive integer >=0 
                               max 64
  --data_type arg (=txt)       Data Type: txt | bin | random
  --input_data arg             Message Input File, full path
  --rs2_inner_tag_pos arg (=2) Concatenated RS Inner tag position, positive 
                               integer >=0
  --rm_order arg (=)          Reed Muller monomial order, positive integer >=0

Python CSV benchmark plot utility:

(.venv)$ python3 ./bin/benchmark_csv_plot.py --help
usage: benchmark_csv_plot.py [-h] [--csv_path CSV_PATH] [--output_plot_path OUTPUT_PLOT_PATH] [--idtypes IDTYPES [IDTYPES ...]] [--strlens STRLENS [STRLENS ...]] [--expsizes EXPSIZES [EXPSIZES ...]] [--tagpos TAGPOS [TAGPOS ...]]

Plot CSV data.

options:
  -h, --help            show this help message and exit
  --csv_path CSV_PATH [CSV_PATH ...]
                        Path to the CSV files. e.g. ./data1.csv ./data2.csv
  --output_plot_path OUTPUT_PLOT_PATH
                        Path to save plots. e.g. ./output.png
  --idtypes IDTYPES [IDTYPES ...]
                        List of ID types. e.g. RSID SHA256ID
  --lengths LENGTHS [LENGTHS ...]
                        List of string lengths. e.g. 10 100 1000 10000
  --expsizes EXPSIZES [EXPSIZES ...]
                        List of experiment sizes. e.g. 8 16
  --tagpos TAGPOS [TAGPOS ...]
                        List of tag positions. e.g. 2 4

Development

• Setup Clang formatter as the default linting tool before committing any changes:

  • Install clang-format tool in your environment. On Ubuntu install clang-format with this command:

    sudo apt install clang-format
    

  • Install VS code extension Clang-Format to add the formatter to list of formatters

    Once Installed, open any *.cpp file and from command selection palette (Ctrl+Shift+P) type Format Document With... -> Configure Default Formatter -> Clang-Format

  • See .clang-format file for applied configuration

• Enable clang formatter to auto apply formatting on save with VS code Add "editor.formatOnSave": true, to your settings.json file

• Setup Python formatter as default linting tool for python before committing any changes

  • Install VS code extension Black Formatter
  • Once installed in VS code enable this setting: Add the following lines to settings.json Ctrl+Shift+P and type Preferences: Open User Settings (JSON): This should enable format on save and set Black Formatter as default

    "[python]": {
        "editor.defaultFormatter": "ms-python.black-formatter",
        "editor.formatOnSave": true
      }
    

• Tips

  To view Make Build when using CMAKE you can add to CMake cacheVariables this option to print more detailed verbose output of makefile for debugging or troubleshooting linking errors and check if the set flags are applied correctly.

  "CMAKE_VERBOSE_MAKEFILE": "ON"
  

Documentation

To generate documentation ensure docs/doxyfile is configured correctly.

• Prerequisites

  Install Doxygen Graphviz and LaTeX on your system. On Ubuntu this can be installed with following:

  sudo apt install doxygen graphviz texlive-latex-extra 
  

• Build Documentation

  From your build directory run the following to generate html and pdf documentation

  make doc
  

  When using CmakePresets.json run the following command from the root directory of repo to generate documentation:

  cmake --build --target doc --preset  <buildPresets.name>
  

• View Documentation

  The html documentation can be viewed in a web browser by opening file docs/html/index.html

  Tip when using WSL the documentation can be viewed by opening the following address in a browser: file://wsl.localhost/Ubuntu/<PATH_TO_IDCODES>/build/docs/html/index.html

  The pdf documentation can be found inside build directory at docs/latex/idcodes_documentation.pdf

CI/CD Pipeline

act - Run GitHub Actions Locally

Install Docker

act uses Docker to run workflows, so ensure Docker is installed on your system. You can install Docker using various means. We will show how to install Docker using apt on Ubuntu as desribed in Install Docker Engine on Ubuntu -> Install using the apt repository

# Uininstall old versions of Docker components
for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

# Install latest version
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

After successfully installing docker follow the steps outlined in Linux Post-Install guide.

Configure Docker to startup automatically on boot with systemd

sudo systemctl enable docker.service
sudo systemctl enable containerd.service

Make sure to restart your system before the next step

Verify installation is successful

sudo docker run hello-world

Post-Install Setup for non-root user, create docker group and add your user to it:

sudo groupadd docker
sudo usermod -aG docker $USER

Now, restart your terminal or reboot the system to ensure changes are applied correctly. Verify by running command docker run hello-world without sudo.

Install Act

Follow the instructions to install act on official website through tar package Copy the link for your package latest release and follow instructions below:

mkdir temp
cd temp
# Replace <arch_tar.gz> in the next command with Linux 64-bit (arm64/aarch64) link below
#https://github.com/nektos/act/releases/latest/download/act_Linux_arm64.tar.gz
#Replace <arch_tar.gz> in the next command with Linux 64-bit (amd64/x86_64) link below
#https://github.com/nektos/act/releases/latest/download/act_Linux_x86_64.tar.gz 
wget <arch_tar.gz>

Untar and install

tar -xvf <arch_tar.gz>.tar.gz 
# Move act binary to /usr/bin/
sudo mv act /usr/bin/
# Delete temporary folder
cd ..
rm -rd temp

Check if act is successfully installed by running:

act --version

To run the default workflow defined in .github/workflows/ci.yml: in jobs section <Job ID> is the name of the build. which can be verified by running act --list

act -j `<Job ID>`

Using -W <path_to_workflows> override default github workflow

# Specify your own workflow for act
act -W '.act/workflows/'

When run first time select Medium image size ~ 500MB

Github Actions Run Github Actions on Cloud

Install Github Actions VS Code Extension to view logs

Runs all workflows by default. The Running actions progress can be seen here: Actions Page

Runs the default workflow defined in .github/workflows/ci.yml: in jobs section

When a push or pull request is made on branch_name as defined in ci.yml all github actions are executed

on:
  push:
    branches:
      - branch_name
  pull_request:
    branches:
      - branch_name

The released artifacts can be found at Release Page.

Building using docker

VS Code extensions to use to see running containers and images.

Docker VS Code Extension

We define settings for container in Dockerfile in current directory . and build image with -t, --tag Name and optionally a tag in the "name:tag" format idcodes:latest using

docker build -t idcodes:latest .

Run Container

docker run idcodes

Open Docker Container in -i, --interactive interactive mode with -t, --tty in a tty console and --rm remove on exit

docker run --rm -i -t idcodes:latest

Build docker container and use it in act GH Action job

# First build the docker container 
docker build -t idcodes .
# Use local container to run jobs on self-hosted container in this case simply the idcodes:latest
act --container-architecture=linux/arm64 --pull=false -P self-hosted=idcodes:latest -W .act/workflows/ -j release

Create Release with GH Actions job

# Install gh cli
sudo apt install gh
# Configure with SSH
gh auth login

# To generate a token run this command
gh auth token

#
mkdir -p $HOME/.act
echo "--artifact-server-path $HOME/.act" >> $HOME/.actrc
# Generate a token by visiting this site: https://github.com/settings/tokens and replace SECRET_VALUE below
echo "GITHUB_TOKEN=SECRET_VALUE" >> ./act/act.secrets
# For secrets use `--secret-file ./act/act.secrets`
# Alternatively use `-s GITHUB_TOKEN="$(gh auth token)"`
docker build --progress=plain -t idcodes_act_node -f dev/docker_templates/act_node/Dockerfile .
# Replace  --container-architecture=linux/arm64 | amd64 appropriately
# Otherwise you will get this Error: Error response from daemon: pull access denied for idcodes_act_node, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
act --use-new-action-cache --verbose --secret-file ./act/act.secrets --container-architecture=linux/amd64 --pull=false -P self-hosted=idcodes_act_node:latest -W .act/workflows/ -j release
act --use-new-action-cache --verbose -s GITHUB_TOKEN="$(gh auth token)" --container-architecture=linux/arm64 --pull=false -P self-hosted=idcodes_act_node:latest -W .act/workflows/ -j release