scramjet-api-client 1.1.1

No project description provided

Scramjet Transform Hub API Client

⭐ Star us on GitHub — it motivates us a lot! 🚀

The package provides an API Client for use with Scramjet Transform Hub.

Usage:

import asyncio
from client.host_client import HostClient
from client_utils.base_client import BaseClient

# your middleware token
token = ''

# set the token
BaseClient.setDefaultHeaders({'Authorization': f'Bearer {token}'})

# middleware url
api_base ='https://api.scramjet.cloud/api/v1' 

# url = {middlewareURL}/space/{manager_id}/api/v1/sth/{host_id}/api/v1
host = HostClient(f'{api_base}/space/org-aa5bu150-9o5c-489b-83e3-b1yf7e086f3h-manager/api/v1/sth/sth-0/api/v1')

res = asyncio.run(host.get_version())
print(res)

More reading

In the link below you will find more information about our stream protocol and API usage.

See the code documentation here: scramjetorg/transform-hub/docs/read-more/stream-and-api.md

Scramjet Transform Hub

This package is part of Scramjet Transform Hub.

Scramjet Transform Hub is a deployment and execution platform. Once installed on a server, it will allow you to start your programs and keep them running on a remote machine. You will be able to start programs in the background or connect to them and see their output directly on your terminal. You will be able to pipe your local data to the program, as if it was running from your terminal. You can start your server in AWS, Google Cloud or Azure, start it on your local machine, install it on a Raspberry Pi or wherever else you'd like.

Use cases

There's no limit what you can use it for. You want a stock checker? A chat bot? Maybe you'd like to automate your home? Retrieve sensor data? Maybe you have a lot of data and want to transfer and wrangle it? You have a database of cities and you'd like to enrich your data? You do machine learning and you want to train your set while the data is fetched in real time? Hey, you want to use it for something else and ask us if that's a good use? Ask us via email or hop on our Scramjet Discord!

---

Host operations

[ GET ] /api/v1/version - show the Host version

Parameters

No parameters

Responses

Successful operation code: 200

{ "version" : "0.12.2" }

[ GET ] /api/v1/sequences - show all Sequences saved on the Host

Parameters

No parameters

Responses

Successful operation code: 200

[
  {
    "instances": [], // a list of all running Instances of this Sequence
    "id": "eea8bc33-440f-4a17-8931-eb22a17d5d56", // Sequence ID
    "config": {
      "container": {
        "image": "scramjetorg/runner:0.12.2",
        "maxMem": 512,
        "exposePortsRange": [
          30000,
          32767
        ],
        "hostIp": "0.0.0.0"
      },
      "name": "@scramjet/hello-alice-out",
      "version": "0.12.2",
      "engines": {
        "node": ">=10"
      },
      "config": {},
      "sequencePath": "index", // a path to file with a main function
      "packageVolumeId": "eea8bc33-440f-4a17-8931-eb22a17d5d56"
    }
  },
  {
    "instances": [
      "02381acf-cb16-4cff-aa9b-f22f04ada94f"
    ],
    "id": "3ec02b93-4ca9-4d23-baab-048dab5ffda4",
    "config": {
      "container": {
        "image": "scramjetorg/runner:0.12.2",
        "maxMem": 512,
        "exposePortsRange": [
          30000,
          32767
        ],
        "hostIp": "0.0.0.0"
      },
      "name": "@scramjet/checksum-sequence",
      "version": "0.12.2",
      "engines": {},
      "config": {},
      "sequencePath": "index.js",
      "packageVolumeId": "3ec02b93-4ca9-4d23-baab-048dab5ffda4"
    }
  }
]

[ GET ] /api/v1/instances - show all Instances running on the Host

Parameters

No parameters

Responses

Successful operation code: 200

[
  {
    "id": "02381acf-cb16-4cff-aa9b-f22f04ada94f", // Instance ID
    "sequence": "3ec02b93-4ca9-4d23-baab-048dab5ffda4" // Sequence ID
  },
  {
    "id": "ab0272d8-c9b0-43f7-9e7e-bcac9ec0f21f",
    "sequence": "e4ca555c-ced1-4a13-b531-f43016eaf4ed"
  }
]

[ GET ] /api/v1/load-check - monitor CPU, memory and disk usage metrics on the Host machine

Parameters

No parameters

Responses

Successful operation code: 200

{
  "avgLoad": 0.08,
  "currentLoad": 5.190776257704936,
  "memFree": 4634816512,
  "memUsed": 8050364416,
  "fsSize": [
    {
      "fs": "/dev/sda1",
      "type": "ext4",
      "size": 20838993920,
      "used": 14939455488,
      "available": 5882761216,
      "use": 71.75,
      "mount": "/"
    },
    {
      "fs": "/dev/sda15",
      "type": "vfat",
      "size": 109422592,
      "used": 9621504,
      "available": 99801088,
      "use": 8.79,
      "mount": "/boot/efi"
    }
  ]
}

[ GET ] /api/v1/log - monitor CPU, memory and disk usage metrics on the Host machine

Parameters

No parameters

Responses

Content-type: application/octet-stream

Successful operation code: 200

2021-11-19T16:04:47.094Z log (object:Host) Host main called.
2021-11-19T16:04:47.100Z info (object:SocketServer) Server on: /tmp/scramjet-socket-server-path
2021-11-19T16:04:47.104Z info (object:Host) API listening on: 127.0.0.1:8000
2021-11-19T16:05:08.228Z info (object:Host) New sequence incoming...
2021-11-19T16:05:08.229Z log (object:LifecycleDockerAdapterSequence) Docker sequence adapter init.
2021-11-19T16:05:08.229Z log (object:DockerodeDockerHelper) Checking image scramjetorg/pre-runner:0.12.2
2021-11-19T16:05:12.234Z info (object:LifecycleDockerAdapterSequence) Docker sequence adapter done.
2021-11-19T16:05:12.246Z log (object:LifecycleDockerAdapterSequence) Volume created. Id:  c50fe4d3-89cc-4685-a82a-16cbc744733d
2021-11-19T16:05:12.246Z log (object:LifecycleDockerAdapterSequence) Starting PreRunner { image: 'scramjetorg/pre-runner:0.12.2', maxMem: 128 }
2021-11-19T16:05:13.536Z log (object:DockerodeDockerHelper) Checking image scramjetorg/runner:0.12.2
2021-11-19T16:05:16.670Z info (object:SequenceStore) New sequence added: c50fe4d3-89cc-4685-a82a-16cbc744733d
2021-11-19T16:05:16.672Z info (object:Host) Sequence identified: {
  container: {
    image: 'scramjetorg/runner:0.12.2',
    maxMem: 512,
    exposePortsRange: [ 30000, 32767 ],
    hostIp: '0.0.0.0'
  },
  name: '@scramjet/multi-outputs',
  version: '0.12.2',
  engines: {},
  config: {},
  sequencePath: 'index.js',
  packageVolumeId: 'c50fe4d3-89cc-4685-a82a-16cbc744733d'
}
2021-11-19T16:05:16.691Z debug (object:Host) Request date: 2021-11-19T16:05:08.239Z, method: POST, url: /api/v1/sequence, status: 202

---

Sequence operations

[ POST ] /api/v1/sequence - add new sequence

Parameters

Name	Type	Description	Required
file	binary	compressed package in tar.gz format	yes
appConfig	json	additional package.json config file	no

Responses

Accepted operation code: 202

{
  "id": "2c3068e5-7c74-45bb-a017-1979c41fc6d0" // sequence id
}

[ GET ] /api/v1/sequences - show list of sequences

Parameters

No parameters

Responses

Successful operation code: 200

[
  {
    "instances": [
      "742d2713-7ab6-4cde-82f3-a7beabdd4e98"       // list of sequence instances
    ],
    "id": "bdef63db-d3a0-45c8-85db-e94ebb96097f",  // sequence id
    "config": {
      "container": {
        "image": "scramjetorg/runner:0.12.2",
        "maxMem": 512
      },
      "name": "@scramjet/transform-hub",
      "version": "0.12.2",
      "engines": {},
      "config": {},
      "sequencePath": "index.js",
      "packageVolumeId": "bdef63db-d3a0-45c8-85db-e94ebb96097f"
    }
  }
]

[ POST ] /api/v1/sequence/:id/start - start chosen sequence

Parameters

Name	Type	Description	Required
appConfig	json	additional package.json config file	no
args	json	additional arguments that instance should starts with	no

Responses

Successful operation code: 200

{
  "id": "681c856e-dfa4-46a1-951d-47b27345552e"
}

[ DELETE ] /api/v1/sequence/:id - delete a sequence by id

Parameters

No parameters

Responses

Successful operation code: 200

{
  "id": "2c3068e5-7c74-45bb-a017-1979c41fc6d0"
}

Conflict operation code: 409 - the instance is still running

{
  "error": "Can't remove sequence in use."
}

---

Instance basic operations

[ GET ] /api/v1/instances - list all instances

Parameters

No parameters

Responses

Successful operation code: 200

[
  {
    "id": "742d2713-7ab6-4cde-82f3-a7beabdd4e98",
    "sequence": "bdef63db-d3a0-45c8-85db-e94ebb96097f"
  },
  {
    "id": "681c856e-dfa4-46a1-951d-47b27345552e",
    "sequence": "bdef63db-d3a0-45c8-85db-e94ebb96097f"
  },
  {
    "id": "21f787ed-6b9e-4e9f-828e-afe428d84833",
    "sequence": "bdef63db-d3a0-45c8-85db-e94ebb96097f"
  }
]

[ GET ] /api/v1/instance/:id - show data of chosen instance

Parameters

No parameters

Responses

Name	Code	Description
Successful operation	200	Returns JSON data
Not Found operation	404	For example if instance was already stopped

{
  "created": "2021-10-29T16:08:36.524Z",
  "started": "2021-10-29T16:08:38.701Z",
  "sequenceId": "b0c02fdc-b05f-4f26-9d68-43a702eb7b44"
}

[ POST ] /api/v1/instance/:id/_stop - end instance gracefully and prolong operations or not for task completion​

Parameters

Name	Type	Description	Required
timeout	number	The number of milliseconds before the Instance will be killed. Default: 7000ms.	no
canCallKeepalive	boolean	If set to true, the instance will prolong the running. Default: false.	no

Responses

Successful operation code: 200

{
   "code": 0,
   "type": "string",
   "message": "string"
}

[ POST ] api/v1/instance/:id/_kill - end instance gracefully waiting for unfinished tasks

Parameters

No parameters

Responses

Accepted operation code: 202

No body returned

[ GET ] /api/v1/instance/:id/health - check status about instance health

Parameters

No parameters

Responses

Successful operation code: 200

{
  "cpuTotalUsage": 529325247,
  "healthy": true,
  "limit": 536870912,
  "memoryMaxUsage": 16117760,
  "memoryUsage": 14155776,
  "networkRx": 1086,
  "networkTx": 0,
  "containerId": "1c993c4ff774fac06185aa9554cf40c23b03e1479a7e0d14827708161b08ae51"
}

---

Instance advanced operation

[ POST ] /api/v1/instance/:id/_event - send event to the Instance

Parameters

Name	Type	Description	Required
eventName	string	Name of an event	true
message	string	JSON formatted event payload	false

Responses

Content-type: application/octet-stream

[ GET ] /api/v1/instance/:id/event - get the data stream with the events from the Instance

Parameters

No parameters

Responses

Content-type: application/octet-stream

[ GET ] /api/v1/instance/:id/once - get the last event sent by the Instance

Parameters

No parameters

Responses

Content-type: application/octet-stream

[ POST ] /api/v1/instance/:id/input - send data to the input stream of the Instance to consume it

Parameters

No parameters

Responses

Name	Code	Description
Successful operation	200	-
Not Acceptable operation	406	Instance expects the input to be provided from the Topic API

[ GET ] /api/v1/instance/:id/output - get stream data from an instance and consume it through the endpoint

Parameters

No parameters

Responses

Content-type: application/octet-stream

[ POST ] /api/v1/instance/:id/stdin​ - process.stdin

Parameters

No parameters

Responses

Successful operation code: 200

[ GET ] /api/v1/instance/:id/stdout - process.stdout

Parameters

No parameters

Responses

Content-type: application/octet-stream

[ GET ] /api/v1/instance/:id/stderr - process.stderr

Parameters

No parameters

Responses

Content-type: application/octet-stream

[ GET ] /api/v1/instance/:id/log - stream all instance logs

Parameters

No parameters

Responses

Content-type: application/octet-stream

Successful operation code: 200

2021-11-19T16:12:22.948Z log (Sequence) 42
2021-11-19T16:12:23.949Z log (Sequence) 41
2021-11-19T16:12:24.950Z log (Sequence) 40
2021-11-19T16:12:25.951Z log (Sequence) 39
2021-11-19T16:12:26.952Z log (Sequence) 38
2021-11-19T16:12:27.952Z log (Sequence) 37
2021-11-19T16:12:28.953Z log (Sequence) 36
2021-11-19T16:12:29.953Z log (Sequence) 35

---

Service Discovery: Topics

If a given topic does not exist, Transform-Hub creates it and stores the sent data in the newly created topic. The data is stored in the topic until the data is not consumed (either by the Topic API or by the Instances subscribing to this topic). When the data are sent to the topic they are written to the returned stream.

[ POST ] /api/v1/topics/:name​ - sends data to the topic

Parameters

No parameters

Request Headers

Header	Type	Description	Default	Required
x-end-stream	boolean	If set to true, then close topic stream after processing the request.	false	no
content-type	text, application	Specifies data type of this topic	application/x-ndjson	no

Supported types: text/x-ndjson, application/x-ndjson, application/x-ndjson, text/plain, application/octet-stream

Responses

Name	Code	Description
Successful operation	200	data was sent with the header indicating the end of data
Successful operation	202	data was sent without the header indicating the end of data (default)

[ GET ] /api/v1/topics/:name​ - get data from the topic

Parameters

No parameters

Responses

Topic data stream.

Successful operation code: 200

{
  "source": "Twitter",
  "id": "850006245121695778",
  "content": "Natural wetlands make up ~30% of global total CH4 emissions",
  "user": {
    "id": 1234994945,
    "name": "Climate Change Conference",
    "screen_name": "Climate Change",
  }
}

---

Some important links

• Scramjet, the company behind Transform Hub
• The Scramjet Framework - functional reactive stream processing framework
• The Transform Hub repo on github
• You can see the Scramjet Transform Hub API docs here
• You can see the CLI documentation here, but si help should also be quite effective.
• Don't forget to ⭐ this repo if you like it, subscribe to releases and keep visiting us for new versions and updates.
• You can open an issue - file a bug report or a feature request here

License and contributions

This module is licensed under AGPL-3.0 license.

The Scramjet Transform Hub project is dual-licensed under the AGPL-3.0 and MIT licenses. Parts of the project that are linked with your programs are MIT licensed, the rest is AGPL.

Contributions

We accept valid contributions and we will be publishing a more specific project roadmap so contributors can propose features and also help us implement them. We kindly ask you that contributed commits are Signed-Off git commit --sign-off.

We provide support for contributors via test cases. If you expect a certain type of workflow to be officially supported, please specify and implement a test case in Gherkin format in bdd directory and include it in your pull request. More info about our BDD test you will find here.

Help wanted 👩‍🎓🧑👱‍♀️

The project need's your help! There's lots of work to do and we have a lot of plans. If you want to help and be part of the Scramjet team, please reach out to us, on discord or email us: [e-mail].

Donation 💸

Do you like this project? It helped you to reduce time spent on delivering your solution? You are welcome to buy us a coffee ☕ Thanks a lot! 😉

You can sponsor us on github

• There's also a Paypal donation link if you prefer that: