Skip to main content

CI Coverage PyPI License

heliotrapi

An API for small fast data analysis jobs at Diamond Light Source. It is effectively a cloud-native open source FaaS (Function as a Service) platform.

[HEE-LEE-OH-TRAY-PEE-EYE]

Helio - Like the sun, a very brilliant light source.

Heliotrope - An pink-purple indigo-like colour.

HeliotrAPI - The API that comes after blue.

heliotrapi exposes an HTTP API to submit analysis jobs, return queued results, and optionally consume messages from RabbitMQ.

Source https://github.com/DiamondLightSource/heliotrapi
PyPI pip install heliotrapi
Docker docker run ghcr.io/diamondlightsource/heliotrapi:latest
Releases https://github.com/DiamondLightSource/heliotrapi/releases

Example Python usage:

from heliotrapi import __version__

print(f"Hello heliotrapi {__version__}")

To start the api server in dev mode on local host:

uvicorn heliotrapi.main:start_api --reload --factory --host 127.0.0.1 --port 8000

or

heliotrapi serve

Overview

The app accepts analysis jobs via HTTP or the client and stores results in a sqlite database (results.db_path) for a configurable time-to-live. On Kubernetes the database lives on a PersistentVolumeClaim, so results survive pod restarts. Jobs can also be ingested from RabbitMQ if rabbitmq.enabled is set.

Sending/Recieving results using the python client

from heliotrapi.client import AnalysisClient

client = AnalysisClient(
    "https://ixx-analysis.diamond.ac.uk",
    username="admin",
    password="password",  # needed to submit analyses, see Authentication below
)

print(client.available_analyses()) #see available analyses

request_id = client.submit("name_of_analysis", examplekwarg1=1, examplekwarg2=2) # <- if kawrgs are invalid it will raise

result = client.get_result() #returns an AnalysisResult basemodel

result = client.get_request_id_result(request_id) #same as above, but specific

print(result)

Sending/Recieving results as an http request

submit jobs to /analyse as a json blob via an HTTP POST request

if want to call the function "double" eg:

def double(number: float | int) -> float:
    """Example analysis that doubles a number."""
    return number * 2

then the json would be sent as a POST request like:

{"analysis_name": "double", "inputs": {"number": 5}}

the server will handle request_id time and created_at, but if you want you can also send it in it's full form and create a uuid and timestamp yourself:

{"analysis_name":"double","inputs":{"number":5},"request_id":"d68de927-79f5-4df3-83d9-d125445c758a","created_at":"2026-05-29T11:47:09.087317"}

via curl the example command would be:

curl -X POST https://ixx-analysis.diamond.ac.uk/analyse -u admin:password -H "Content-Type: application/json" -d '{"analysis_name":"double","inputs":{"number":5}}'

and to get the last result from /results/latest as a GET request

curl -X GET https://i15-1-analysis.diamond.ac.uk/results/latest

If you are getting redirected due to the a proxy use curl -LX ...

Authentication

Submitting analyses, streaming, and reading results (/results/...) require HTTP Basic auth. The health check, /get_analyses and /endpoints are open.

Auth is on by default. Turn it off in the config with:

auth:
  enabled: false

The username and password are read from the HELIOTRAPI_USERNAME and HELIOTRAPI_PASSWORD environment variables. Never put them in a config file. If they aren't set, they default to admin / password for local development, and the server logs a warning on startup. On Kubernetes they come from a SealedSecret, see Auth credentials.

  • Python client: pass username= and password= to AnalysisClient
  • curl: add -u username:password
  • WebUI: the browser asks for the username and password when the page loads

Basic auth sends the credentials with every request, so only expose the API over HTTPS.

Using the WebUI

You can also navigate to the url or the ip address to be met with:

Web UI

Request flow

  • AnalysisClient submits jobs to /analyse

  • Jobs are queued in QueueManager

  • Workers process jobs in FIFO order

  • Results are returned via /results/id/{request_id} or /results/latest

  • Optional RabbitMQ listener can enqueue jobs automatically

                   AnalysisClient ─────--────────────────
                      │ ▲                │              │
                      ▼ │                ▼              ▼
      Analysis <-── heliotrapi ──---►  RabbitMQ ──---► Results
         Job   ─---►    ▲                │
                        │                │
                        │                │   
                  RabbitListener <───────
    

Using your plugin code

In the config or the helm chart you can changed the following lines:

    plugins:
      paths:
        - ./plugins # local folder for plugins
      github_repos: # list of Https GitHub repos with analysis code (ending with .git)
        - https://github.com/DiamondLightSource/xrpd-toolbox.git
      register_all: False # whether to register all analyses found in plugins or only those decorated

In this example plugins will be downloaded and installed to ./plugins. The github repo https://github.com/DiamondLightSource/xrpd-toolbox.git will be cloned to the .plugins folder.

To add functions that are in your plugin repo, you can use the analysis decorator:

import math

from heliotrapi import analysis

@analysis
def my_custom_func(first_number: float, second_number: float) -> float:

    return math.log(first_number) / math.log(second_number)

Functions should be be strongly typed and return something. By using @analysis in the plugin repo, and decorating your function, heliotrapi will register this as an available function.

Alternatively, you can set register_all to True all of them will be added to heliotrapi's available analyses. All function in the repo will be added to available analyses, with the exceptions of those prefixed with _ or test_. This is not recommended for production use, but is useful for testing.

Kubernetes deployment

This repository includes a Helm chart under ./helm/heliotrapi.

Config support

The service supports configuration from one of these sources:

  • CONFIG_PATH environment variable
  • mounted config file at /etc/config/config.yaml
  • local config.yaml file in the current working directory

In Kubernetes, the Helm chart mounts config.yaml from a ConfigMap and sets:

env:
  - name: CONFIG_PATH
    value: "/etc/config/config.yaml"

RabbitMQ config

The Helm values now expose RabbitMQ settings in the same shape as the app expects:

config:
  rabbitmq:
    enabled: true
    host: ixx-analysis.diamond.ac.uk
    username: guest
    password: guest
    port: 61613
    destinations:
      - "/topic/public.worker.event"
      - "/topic/gda.messages.scan"
      - "/topic/gda.messages.processing"
      - "/topic/public.analysis.trigger"

Auth credentials

When config.auth.enabled is true (the default), the pod reads HELIOTRAPI_USERNAME and HELIOTRAPI_PASSWORD from the Secret named by authSecretName (default heliotrapi-auth). Create it with a SealedSecret while logged in to the target cluster and namespace:

kubectl create secret generic heliotrapi-auth --dry-run=client -o yaml \
  --from-literal=username='<username>' --from-literal=password='<password>' \
  | kubeseal -o yaml > heliotrapi-auth-sealed.yaml

kubectl create -f heliotrapi-auth-sealed.yaml

heliotrapi-auth-sealed.yaml is encrypted, so it's safe to commit. Never commit the plain Secret. Create the SealedSecret before deploying - the pod won't start until the Secret exists.

Helm usage

  1. Build and push your Docker image
podman build -t ghcr.io/diamondlightsource/heliotrapi:latest .
podman push ghcr.io/diamondlightsource/heliotrapi:latest
  1. Render the chart
helm template heliotrapi ./helm/heliotrapi
  1. Dry-run validation
helm template heliotrapi ./helm/heliotrapi | kubectl apply --dry-run=client -f -
  1. Install the chart
helm install heliotrapi ./helm/heliotrapi
  1. Verify the deployment
kubectl get pods
kubectl get svc
  1. Test the API
kubectl port-forward svc/heliotrapi 8000:8000

Then open:

http://localhost:8000/

Example Kubernetes Deployment

Go to the i15-services repo on gitlab.diamond.ac.uk to see an example of how to set it up

Create a secret:

  1. Go to the right cluster and namespace. Do:
module load k8s-ixx
  1. Type your password. Nothing shows while you type:
read -rsp "Password: " PW; echo
  1. Create the sealed secret (change ixx-heliotrapi to your beamline):
kubectl create secret generic heliotrapi-auth --dry-run=client -o yaml --from-literal=username='ixx-heliotrapi' --from-literal=password="$PW" | kubeseal -o yaml | kubectl create -f -
  1. Clear the password from your shell:
unset PW
  1. Check it. Both a sealedsecret and a secret should be listed:
kubectl get sealedsecret,secret heliotrapi-auth
  1. Now add to the blueapi config:
env:
  - name: HELIOTRAPI_USERNAME
    valueFrom:
      secretKeyRef:
        name: heliotrapi-auth
        key: username
  - name: HELIOTRAPI_PASSWORD
    valueFrom:
      secretKeyRef:
        name: heliotrapi-auth
        key: password
  1. Deploy. Restart the pod. Go to the webpage and use ixx-heliotrapi as the username and your password as the password to see results on the frontend

8?. If you need to delete the secret and re-do it:

kubectl delete sealedsecret heliotrapi-auth

Metadata

Release files for heliotrapi 0.4.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for heliotrapi 0.4.2
File Size Uploaded Provenance Actions
heliotrapi-0.4.2.tar.gz 781.4 kB Without provenance Details for heliotrapi-0.4.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for heliotrapi 0.4.2
File Interpreter ABI Platform Actions
heliotrapi-0.4.2-py3-none-any.whl Python 3 none any Details for heliotrapi-0.4.2-py3-none-any.whl

Total release size: 840.9 kB

Release files / heliotrapi-0.4.2.tar.gz

Download URL heliotrapi-0.4.2.tar.gz
Size 781.4 kB
Tags Source
SHA-256 checksum
How to use checksums
f1210b784938c3062c615c78f96004cd89b27ab53b8b9cc6ce5032e2178be338
BLAKE2b-256 checksum
How to use checksums
48a6fbd354962e2b23f2276f104d4e626c2237a532aefd9c14a12e5f76f7a4da
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / heliotrapi-0.4.2-py3-none-any.whl

Download URL heliotrapi-0.4.2-py3-none-any.whl
Size 59.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
ca7ef41f51eb2fcc874ad895970025b6087d3379fffab5a13e4479792a8dac94
BLAKE2b-256 checksum
How to use checksums
a52c812e8cebfa6c57cd4664ea589683fa662f824d25a3199cd6845acfba726e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Security for heliotrapi 0.4.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.4.2 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page