Introduction

Nexus SDK Py is a Python development kit for Nexus client applications. It builds upon Golang Client via cgo. Python SDK does not use any Python-level HTTP middleware for Nexus interactions, but authentication might require it.

SDK is tested against a Nexus stack in a docker-compose deployment, backed by kind Kubernetes clusters.

Quickstart

Install CGO library from Go SDK by running:

chmod +x ./sdk-installer.sh
./sdk-installer.sh

Setting up the development environment

To set up the development environment, follow these steps:

To start a local Nexus stack for testing, run:

just up

The command will start the Kind cluster and deploy Nexus stack on it. You can then run tests against this local deployment.

The Nexus Scheduler API will be available at http://localhost:5555/scheduler and the Nexus Receiver API will be available at http://localhost:8080/receiver. Note that the cluster's ingress configured to rewrite URL paths to reduce number of ports you need to interact with and reduce probability of errors related to ports collision. For example, when you send a request to http://localhost:5555/scheduler/api/something/something, it will be automatically rewritten to http://scheduler-pod:8080/api/something/something and forwarded to the Scheduler API.

The Scylla DB will be available at localhost:9042 and you can connect to it using any CQL client. The default credentials are cassandra/cassandra.

The MinIO s3 API will be available at localhost:9000 and you can connect to it using any S3 client. The default credentials are minioadmin/minioadmin. The minio console is not exposed, but you can use the mc client to interact with it.

In case you are testing changes for Go SDK, clone branch you are testing and compile the .so file from source:

go build -v -buildmode=c-shared -o nexus_sdk.so main.go

Afterwards, copy the nexus_sdk.so under nexus_client_sdk/.extensions/nexus_sdk.so.

Initialize a client and retrieve results for a tagged submission:

from nexus_client_sdk.models.access_token import AccessToken
from nexus_client_sdk.models.scheduler import SdkCustomRunConfiguration
from nexus_client_sdk.clients.nexus_scheduler_client import NexusSchedulerClient

token = "..."
client = NexusSchedulerClient.create("https://localhost:8080", lambda: AccessToken.empty())

alg_params = {"field1": {"field2": 1, "field3": "abc"}, "field4": "cde"}

# create a run
new_run = client.create_run(
    algorithm_parameters=alg_params,
    algorithm_name="test-algorithm",
    custom_configuration=SdkCustomRunConfiguration.create(version="v1.2.3"),
    tag="test-py-sdk",
    payload_valid_for="6h",
)

print(f"Run id: {new_run}")

for result in client.get_run_results("abc"):
    print(result)

Nexus Development Framework

Apart from API clients for Nexus, SDK ships a development framework under nexus subpackage. It allows to create production-grade, asyncio-native ML/AI solutions that use a unified structure and are compose of objects and object relations, rather than methods. Nexus turns ML/AI apps into standard Python applications and removes the common noise found in notebook- type code, such as variable reassignment, frequent data copying due to lack of reusable code, copy-paste of code etc.

Nexus's design makes life even easier when using AI code generation, as it is essentially a framework an AI agent can follow to generate a working data science pipeline. Nexus takes care of result accounting, error handling, logging, metric reporting and, most importantly, execution flow. A key feature in Nexus is automatic resolution of execution graph via dependency injection. In essence, a developer just needs to specify which inputs are required for an algorithm to run, and provide class implementations for this, and Nexus will take care of the rest. This also implies that whether an IO operation happens, such as a database read or a file load, Nexus will utilize asyncio coroutines to run multiple IO ops in parallel, significantly increasing the execution speed, without any need for a developer to understand async programming.

For a example of how to use Nexus, take a look at a Sample Algorithm and a corresponding test configuration and a test itself.

Execution tree

Nexus provides a set of utilities that allow viewing and inspecting the execution tree:

from nexus_client_sdk.nexus.execution.trees import get_tree
from tests.algorithms.minimalistic.minimalistic_algorithm_sample import TestMinimalisticAlgorithm

print(get_tree(TestMinimalisticAlgorithm).serialize())

# graph TB
# TESTMINIMALISTICALGORITHM["TestMinimalisticAlgorithm"] --> XYPROCESSOR["XYProcessor"] --> XYREADER["XYReader"]
# TESTMINIMALISTICALGORITHM["TestMinimalisticAlgorithm"] --> ZPROCESSOR["ZProcessor"] --> ZREADER["ZReader"]

Handling Compressed Payloads

Nexus supports reading compressed payloads for efficient data transfer. When a payload is compressed, it must include both the compressed content and a reference to the decompression function.

Payload Structure

A compressed payload should be a json with the following keys:

  • content: The compressed data (as a base64-encoded string).
  • decompressor_import_path: The Python import path to the decompression function.

Example:

{
    "content": "SGVsbG8gd29ybGQ=",  # base64-encoded string of compressed bytes
    "decompressor_import_path": "my_module.my_decompress"
}

When Nexus receives such a payload, it will:

  1. Base64-decode the content field to obtain the compressed bytes.
  2. Dynamically import and call the function specified by decompressor_import_path to decompress the payload.
  3. Use the decompressed data as the actual payload for the algorithm.

This mechanism allows for flexible, pluggable decompression logic, as long as the function path is importable and callable in the runtime environment.

Automatic Payload Compression

Nexus can automatically compress and decompress payloads when using RemoteAlgorithm. To use this feature, you must first configure it with environment variables and then explicitly enable it in your RemoteAlgorithm implementation.

Step 1: Configuration (Environment Variables)

First, you need to provide the Python import paths for your compression and decompression logic. Setting these environment variables allows Nexus to create an injectable Compressor service.

  • NEXUS__REMOTE_ALGORITHM__COMPRESSION_IMPORT_PATH: The import path to your compression function (e.g., my_module.my_compress).
  • NEXUS__REMOTE_ALGORITHM__DECOMPRESSION_IMPORT_PATH: The import path to your decompression function (e.g., my_module.my_decompress).

Step 2: Enabling Compression in Your Algorithm

Once the environment variables are set, you can activate compression on a RemoteAlgorithm instance by providing two arguments during its initialization:

  1. compress_payload=True: This boolean flag signals your intent to use compression for this remote algorithm.
  2. compressor=<injected_compressor_instance>: You must inject the Compressor service that Nexus creates from your environment variables.

Important Requirement

For compression to work, both conditions must be met. The application will raise an error if compress_payload is set to True but a valid Compressor instance is not injected. Ensure that the required environment variables are set so the Compressor service can be created and injected successfully.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.7.2)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for esddev from gravatar.com esddev

Credits

PyPI data
Data sourced directly from PyPI's database.

Author/Maintainer: ECCO Sneaks & Data

License

Apache Software License (Apache-2.0)

Requires

Python <3.13, >=3.11

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

nexus_client_sdk-1.7.1-cp311-cp311-manylinux_2_28_x86_64.whl (25.1 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.28+ x86-64

nexus_client_sdk-1.7.1-cp311-cp311-manylinux_2_28_aarch64.whl (23.0 MB view details)

Uploaded CPython 3.11manylinux: glibc 2.28+ ARM64

nexus_client_sdk-1.7.1-cp311-cp311-macosx_13_0_arm64.whl (11.4 MB view details)

Uploaded CPython 3.11macOS 13.0+ ARM64

File details

Details for the file nexus_client_sdk-1.7.1-cp311-cp311-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for nexus_client_sdk-1.7.1-cp311-cp311-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 22faafad2cfcea2b8320a989c5bc11468a6f9479c4e1371587f51cf64ef81234
MD5 0c1f5c2711f52a00af084b71c3048827
BLAKE2b-256 cd1b7a51e90262267c785ac049314b95bd413018c811115c8b448e745d6e92f4

See more details on using hashes here.

File details

Details for the file nexus_client_sdk-1.7.1-cp311-cp311-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for nexus_client_sdk-1.7.1-cp311-cp311-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 776159b2511e1374a65bf264d2e12f9edf22e440034621417112cb17f4ba934c
MD5 412a7f600854e8538861f3a2abab6bec
BLAKE2b-256 3b9b31a5e93b044a1a1ac036f8a8404664ae4fc5629d6660881bb5c1e40b513d

See more details on using hashes here.

File details

Details for the file nexus_client_sdk-1.7.1-cp311-cp311-macosx_13_0_arm64.whl.

File metadata

File hashes

Hashes for nexus_client_sdk-1.7.1-cp311-cp311-macosx_13_0_arm64.whl
Algorithm Hash digest
SHA256 2ac445e1de4bfc6add4cc5275842ec7d6cb266a52207aabf71cdddd4f7360a18
MD5 ed5536c3c9cbd74db77e52b6c2c6f14d
BLAKE2b-256 edf51d87f2fa732fa5c95f68174ed2d2f4415dd02d412febcdd7cf3f01f8ddf5

See more details on using hashes here.

Release history Release notifications | RSS feed

1.7.2

3 files

This release

1.7.1 This release

3 files

1.7.0

3 files

1.6.9

3 files

1.6.8

3 files

1.6.7

3 files

1.6.6

3 files

1.6.5

3 files

1.6.4

3 files

1.6.3

3 files

1.6.2

3 files

1.6.1

3 files

1.6.0

3 files

1.5.8

3 files

1.5.7

3 files

1.5.6

3 files

1.5.5

3 files

1.5.4

3 files

1.5.3

3 files

1.5.2

3 files

1.5.1

3 files

1.5.0

3 files

1.4.8

3 files

1.4.6

3 files

1.4.5

3 files

1.4.4

3 files

1.4.3

3 files

1.4.2

3 files

1.4.1

3 files

1.4.0

3 files

1.3.8

3 files

1.3.7

3 files

1.3.6

3 files

1.3.5

3 files

1.3.4

3 files

1.3.3

3 files

1.3.2

3 files

1.3.1

3 files

1.3.0

3 files

1.2.16

3 files

1.2.15

3 files

1.2.14

3 files

1.2.13

3 files

1.2.12

3 files

1.2.11

3 files

1.2.10

3 files

1.2.9

3 files

1.2.8

3 files

1.2.7

3 files

1.2.6

3 files

1.2.5

3 files

1.2.4

3 files

1.2.3

3 files

1.2.2

3 files

1.2.1

3 files

1.2.0

3 files

1.1.2

3 files

1.1.1

3 files

1.1.0

3 files

1.0.0

3 files

0.3.7

3 files

0.3.6

3 files

0.3.5

3 files

0.3.4

3 files

0.3.3

3 files

0.3.2

3 files

0.3.1

3 files

0.3.0

3 files

0.2.1

3 files

0.2.0

3 files

0.1.1

3 files

0.1.0

3 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page