HOBOLink API client with HDF5/HSDS integration
Project description
HOBO-request
A Kubernetes-ready container application for integrating HOBOlink loggers with HSDS. This application was developed in partnership with the HDF Group with NSF support for the UVA-ARC project (grant #2022639).
Getting started
HOBO-request provides an interface to HOBOlink loggers via a public API that is provided by Onset. It can be used with HSDS for the purposes of large-scale environmental data analysis, or as a stand-alone tool for the purposes of testing. It can also be easily extended to be used with different APIs and sensor networks.
You can install the HOBO-request package or git clone this repository. For installing the pip package, just create a virtual environmental and run:
$ python3 -m venv hobo-request
$ source hobo-request/bin/activate
$ pip install hoborequest
You can also take the development path way and work directly with the source code.
In order to integrate HOBO-request with data loggers, a few libraries / modules need to be installed.
You can proceed by installing the dependencies described in requirements.txt
.
$ python3 -m venv hobo-request
$ source hobo-request/bin/activate
$ pip install -r requirements.txt
HOBO-request is now ready to be used, but first it must be configured!
Configuration
First, copy conf/hobo-connect.conf-SAMPLE
to conf/hobo-connect.conf
.
Then, edit 'conf/hobo-connect.conf' and set the following parameters:
user_id
: your Onset user IDclient_id
: your Onset client IDclient_secret
: API secret key obtained from Onsetstart_date_time
: for precise time slicing, the start time in UTC, format: 2012-01-01 23:00:00end_date_time
: same as above, the end time in UTC, format: 2021-01-12 11:00:00polling_interval_minutes
: if you want to set a regular polling interval, leave start & end time fields empty. Alternatively you can also set a start time, but leave end time empty
Please observe that timestamps are always assumed to be in UTC. For the purposes of data storage, it is more convenient to avoid timezones and daylight-saving differences, and leave to the analyst to manipulate timestamps as he/she/they see fit.
In addition to the configuration of your Onset credentials, you also need to set your HSDS parameters:
hsds_filename
: your HDF data store namehsds_endpoint
: your HSDS endpoint
Please observe that you can also use HOBO-request locally without HSDS for testing purposes. In that case, you only
need to set hsds_filename
to point to the h5 file in your local filesystem.
Finally, you must set the repository where your logger and sensor metadata are registered. This is a crucial step because HOBO-request depends on this config to update your metadata attributes for you. You must set the following fields:
meta_repo
: git repository that contains your deployment metadata filesmeta_local_dir
: local directory where you want to store the repo temporarilymeta_root_path
: repository directory where the HDF root file metadata file ismeta_loggers_dir
: repository directory where the loggers metadata files aremeta_sensors_dir
: repository directory where the sensors metadata files are
Once this is done, we can move on to the generation of your data store.
Creating your HDF data store with metadata
HOBO-request reads YAML files with metadata organized in three levels:
HDF root metadata
(attributes)HOBO loggers
(attributes)HOBO sensors
(attributes)
It is necessary, therefore, to create YAML files following the scheme we describe in the conf/
and
conf/sensors/templates
directories. For example:
uva-arc.yaml
: provides an example of how to set root attributes in your HDF fileloggers/
: provides examples for the configuration of HOBOLink loggers / stationssensors/templates
: provides examples for the configuration of HOBOLink sensors
Once you are done creating your YAML files, you can load them in your newly created HDF file.
Change directory to src/
, then run hobo-config.py
:
To generate your HDF file with root attributes:
./hobo-config.py test.h5 ../conf/uva-arc.yaml
Or, if you are using the pip package, you can just run:
$ hoboconfig test.h5 ../conf/uva-arc.yaml
Please note that you have to change the PATH for the YAML file in the command above. You can find real examples for stations and sensors in this repository.
To load your logger and sensor metadata, you can type:
./hobo-config.py test.h5 /tmp/deployment-metadata/alaska/loggers/*
./hobo-config.py test.h5 /tmp/deployment-metadata/alaska/sensors/*
Or... if you are using the pip package hoborequest
:
$ hoboconfig test.h5 /tmp/deployment-metadata/alaska/loggers/*
$ hoboconfig test.h5 /tmp/deployment-metadata/alaska/sensors/*
Please note that it is necessary to run this step only once when creating the HDF data store.
After you load the HDF file onto HSDS, HOBO-request will automatically pull all the changes
from the git
repository containing your YAML files and load all the metadata information
for you. There is no need to use hobo-config
manually again.
Running HOBO-request to consume sensor data
Now that everything is in place, you have the choice of running HOBO-request locally for tests using a regular HDF file (and h5py) or run the HDF data store on HSDS (with h5pyd). This is done automagically for you, so you just have to point in your config file if you want to use a regular file or interface with HSDS to feed data to your HDF data store.
You can start HOBO-request with a simple command:
$ ./hobo-connect.py
Or, if you are running the pip package:
$ hoboconnect
Et... voilà! The application will return the following message as soon as it is done processing the data:
[HOBO-connect]: data processing completed
Generating a container for HOBO-request
We have provided scripts and deployment files to generate a container for HOBO-request, plus documentation on how to run HSDS on k3s (or microk8s) for local testing.
You can run the following commands to build your container and run it:
$ ./build.sh
$ ./docker_run.sh
The directory k8s/
contains the deployment files and scripts for running HOBO-request on k8s or microk8s.
May the source be with you in your environmental studies!
Licensing
See LICENSE and AUTHORS files for details.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for hoborequest-0.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 438e3d2090cccedad0af0ba27f445fa77c954108ba7f5f99a4cd159f0daf1bca |
|
MD5 | 01af54b9bf1fe1efff85dc9814ad2f90 |
|
BLAKE2b-256 | 3b33b7940859e2ef7c05ff3382b55093cf7e472936d0944eef244e693780b3ae |