heliotrapi
An API for small fast data analysis jobs at Diamond Light Source. It is effectively a cloud-native open source FaaS (Function as a Service) platform.
[HEE-LEE-OH-TRAY-PEE-EYE]
Helio - Like the sun, a very brilliant light source.
Heliotrope - An pink-purple indigo-like colour.
HeliotrAPI - The API that comes after blue.
heliotrapi exposes an HTTP API to submit analysis jobs, return queued results, and optionally consume messages from RabbitMQ.
| Source | https://github.com/DiamondLightSource/heliotrapi |
|---|---|
| PyPI | pip install heliotrapi |
| Docker | docker run ghcr.io/diamondlightsource/heliotrapi:latest |
| Releases | https://github.com/DiamondLightSource/heliotrapi/releases |
Example Python usage:
from heliotrapi import __version__
print(f"Hello heliotrapi {__version__}")
To start the api server in dev mode on local host:
uvicorn heliotrapi.main:start_api --reload --factory --host 127.0.0.1 --port 8000
or
heliotrapi serve
Overview
The app accepts analysis jobs via HTTP or the client and stores results in memory for a configurable time-to-live. Jobs can also be ingested from RabbitMQ if rabbitmq.enabled is set.
Sending/Recieving results using the python client
from heliotrapi.client import AnalysisClient
client = AnalysisClient(
"https://ixx-analysis.diamond.ac.uk",
username="admin",
password="password", # needed to submit analyses, see Authentication below
)
print(client.available_analyses()) #see available analyses
request_id = client.submit("name_of_analysis", examplekwarg1=1, examplekwarg2=2) # <- if kawrgs are invalid it will raise
result = client.get_result() #returns an AnalysisResult basemodel
result = client.get_request_id_result(request_id) #same as above, but specific
print(result)
Sending/Recieving results as an http request
submit jobs to /analyse as a json blob via an HTTP POST request
if want to call the function "double" eg:
def double(number: float | int) -> float:
"""Example analysis that doubles a number."""
return number * 2
then the json would be sent as a POST request like:
{"analysis_name": "double", "inputs": {"number": 5}}
the server will handle request_id time and created_at, but if you want you can also send it in it's full form and create a uuid and timestamp yourself:
{"analysis_name":"double","inputs":{"number":5},"request_id":"d68de927-79f5-4df3-83d9-d125445c758a","created_at":"2026-05-29T11:47:09.087317"}
via curl the example command would be:
curl -X POST https://ixx-analysis.diamond.ac.uk/analyse -u admin:password -H "Content-Type: application/json" -d '{"analysis_name":"double","inputs":{"number":5}}'
and to get the last result from /results/latest as a GET request
curl -X GET https://i15-1-analysis.diamond.ac.uk/results/latest
If you are getting redirected due to the a proxy use curl -LX ...
Authentication
Submitting analyses, streaming, and reading results (/results/...) require HTTP Basic auth.
The health check, /get_analyses and /endpoints are open.
Auth is on by default. Turn it off in the config with:
auth:
enabled: false
The username and password are read from the HELIOTRAPI_USERNAME and HELIOTRAPI_PASSWORD
environment variables. Never put them in a config file. If they aren't set, they default to
admin / password for local development, and the server logs a warning on startup.
On Kubernetes they come from a SealedSecret, see Auth credentials.
- Python client: pass
username=andpassword=toAnalysisClient - curl: add
-u username:password - WebUI: the browser asks for the username and password when the page loads
Basic auth sends the credentials with every request, so only expose the API over HTTPS.
Using the WebUI
You can also navigate to the url or the ip address to be met with:
Request flow
-
AnalysisClient submits jobs to
/analyse -
Jobs are queued in
QueueManager -
Workers process jobs in FIFO order
-
Results are returned via
/results/id/{request_id}or/results/latest -
Optional RabbitMQ listener can enqueue jobs automatically
AnalysisClient ─────--──────────────── │ ▲ │ │ ▼ │ ▼ ▼ Analysis <-── heliotrapi ──---► RabbitMQ ──---► Results Job ─---► ▲ │ │ │ │ │ RabbitListener <───────
Using your plugin code
In the config or the helm chart you can changed the following lines:
plugins:
paths:
- ./plugins # local folder for plugins
github_repos: # list of Https GitHub repos with analysis code (ending with .git)
- https://github.com/DiamondLightSource/xrpd-toolbox.git
register_all: False # whether to register all analyses found in plugins or only those decorated
In this example plugins will be downloaded and installed to ./plugins. The github repo https://github.com/DiamondLightSource/xrpd-toolbox.git will be cloned to the .plugins folder.
To add functions that are in your plugin repo, you can use the analysis decorator:
import math
from heliotrapi import analysis
@analysis
def my_custom_func(first_number: float, second_number: float) -> float:
return math.log(first_number) / math.log(second_number)
Functions should be be strongly typed and return something. By using @analysis in the plugin repo, and decorating your function, heliotrapi will register this as an available function.
Alternatively, you can set register_all to True all of them will be added to heliotrapi's available analyses. All function in the repo will be added to available analyses, with the exceptions of those prefixed with _ or test_. This is not recommended for production use, but is useful for testing.
Kubernetes deployment
This repository includes a Helm chart under ./helm/heliotrapi.
Config support
The service supports configuration from one of these sources:
CONFIG_PATHenvironment variable- mounted config file at
/etc/config/config.yaml - local
config.yamlfile in the current working directory
In Kubernetes, the Helm chart mounts config.yaml from a ConfigMap and sets:
env:
- name: CONFIG_PATH
value: "/etc/config/config.yaml"
RabbitMQ config
The Helm values now expose RabbitMQ settings in the same shape as the app expects:
config:
rabbitmq:
enabled: true
host: ixx-analysis.diamond.ac.uk
username: guest
password: guest
port: 61613
destinations:
- "/topic/public.worker.event"
- "/topic/gda.messages.scan"
- "/topic/gda.messages.processing"
- "/topic/public.analysis.trigger"
Auth credentials
When config.auth.enabled is true (the default), the pod reads HELIOTRAPI_USERNAME and
HELIOTRAPI_PASSWORD from the Secret named by authSecretName (default heliotrapi-auth).
Create it with a SealedSecret
while logged in to the target cluster and namespace:
kubectl create secret generic heliotrapi-auth --dry-run=client -o yaml \
--from-literal=username='<username>' --from-literal=password='<password>' \
| kubeseal -o yaml > heliotrapi-auth-sealed.yaml
kubectl create -f heliotrapi-auth-sealed.yaml
heliotrapi-auth-sealed.yaml is encrypted, so it's safe to commit. Never commit the plain Secret.
Create the SealedSecret before deploying - the pod won't start until the Secret exists.
Helm usage
- Build and push your Docker image
podman build -t ghcr.io/diamondlightsource/heliotrapi:latest .
podman push ghcr.io/diamondlightsource/heliotrapi:latest
- Render the chart
helm template heliotrapi ./helm/heliotrapi
- Dry-run validation
helm template heliotrapi ./helm/heliotrapi | kubectl apply --dry-run=client -f -
- Install the chart
helm install heliotrapi ./helm/heliotrapi
- Verify the deployment
kubectl get pods
kubectl get svc
- Test the API
kubectl port-forward svc/heliotrapi 8000:8000
Then open:
http://localhost:8000/
Example Kubernetes Deployment
Go to the i15-services repo on gitlab.diamond.ac.uk to see an example of how to set it up
Create a secret:
- Go to the right cluster and namespace. Do:
module load k8s-ixx
- Type your password. Nothing shows while you type:
read -rsp "Password: " PW; echo
- Create the sealed secret (change ixx-heliotrapi to your beamline):
kubectl create secret generic heliotrapi-auth --dry-run=client -o yaml --from-literal=username='ixx-heliotrapi' --from-literal=password="$PW" | kubeseal -o yaml | kubectl create -f -
- Clear the password from your shell:
unset PW
- Check it. Both a sealedsecret and a secret should be listed:
kubectl get sealedsecret,secret heliotrapi-auth
- Now add to the blueapi config:
env:
- name: HELIOTRAPI_USERNAME
valueFrom:
secretKeyRef:
name: heliotrapi-auth
key: username
- name: HELIOTRAPI_PASSWORD
valueFrom:
secretKeyRef:
name: heliotrapi-auth
key: password
- Deploy. Restart the pod. Go to the webpage and use ixx-heliotrapi as the username and your password as the password to see results on the frontend
8?. If you need to delete the secret and re-do it:
kubectl delete sealedsecret heliotrapi-auth
Metadata
Release files for heliotrapi 0.4.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| heliotrapi-0.4.1.tar.gz | 779.2 kB | Without provenance | Details for heliotrapi-0.4.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| heliotrapi-0.4.1-py3-none-any.whl | Python 3 | none | any | Details for heliotrapi-0.4.1-py3-none-any.whl |
Total release size: 837.2 kB
Release files / heliotrapi-0.4.1.tar.gz
| Download URL | heliotrapi-0.4.1.tar.gz |
|---|---|
| Size | 779.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
fae0b2c59aec63514bf8da247ad4bc52fd5940952d5870bcaee0e7d7359f7018
|
|
BLAKE2b-256 checksum How to use checksums |
bedd37c347353a88f827636418d43c3a4617232bd7a82f63cfdac03f7ce3ec04
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Release files / heliotrapi-0.4.1-py3-none-any.whl
| Download URL | heliotrapi-0.4.1-py3-none-any.whl |
|---|---|
| Size | 58.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
44e630f4d95153a9e01aca99c6bdabbf16c19c33cd6f3d210c1069af7752131b
|
|
BLAKE2b-256 checksum How to use checksums |
c7d0893af291d223bfecd973351f51452604b184d37d3319a59276d28a526f4b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Security for heliotrapi 0.4.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.