scystream-sdk 1.5.0

The official SDK for developing scystream compute blocks

scystream-sdk

Installation

You can install the package via pip once it's published:

pip install scystream-sdk

Introduction

One of the central concepts of scystream are the so-called Compute Blocks.

A Compute Block describes an independent programm, that acts as some kind of worker which will be scheduled using the scystream-core application. This worker executes a task (e.g. a NLP task, a crawling task).

This SDK aims to provide helper functions and all other requirements you need to implement a custom Compute Block on your own.

Each worker can have multiple entrypoints, each aiming to solve one task. These entrypoints can be configured from the outside using the Settings. These are basically ENV-Variables, which will be parsed & validated using pydantic.

Find the whole Documentation here!

Development of the SDK

Installation

1. Create a venv and use it

python3 -m venv .venv
source .venv/bin/activate

2. Install the package within the venv

[!NOTE] This will also install all the install_requirements from the setup.py

pip install -e .

3. Install the dev dependencies

pip install -e ".[dev]"

4. Develop!

Tests

To run all the tests run the following command:

python3 -m unittest discover -s tests