Python Package providing a suite of processing tools and utilities for Hilltop hydrological data.
Free software: GNU General Public License v3
Documentation: https://hydrobot.readthedocs.io.
Features
Processes data downloaded from Hilltop Server
Creates output files in a format appropriate for returning data to Hilltop
Uses annalist to record changes to data generated from processing
Capable of various automated processing techniques, including:
Clipping data
Removing spikes based on FBEWMA smoothing
Identifying and removing ‘flatlining’ data, where an instrument repeats it’s last collected data point (NOTE: Low precision data may be falsely flagged by this)
Identifying gaps and gap lengths and closing small gaps
Aggregating check data from various sources
Collating comments + metadata from various inspections and related activities
Quality coding data based on NEMS standards
Plotting data, including:
Processed data with quality codes
Comparing raw data to processed data
Showing all changes to the data
Visualizing check points from various sources
Usage (Alpha)
Hydrobot supports a “hybrid” workflow. This means that a large number of sites can be processed automatically, but if on examination an individual site has a problem an adjustment can be applied before the data loaded. This is required for environmental data as there are many various issues that the data can have.
To support this hybrid workflow, Hydrobot processes data on a “one site, one script” basis. All the logic required for an individual site’s processing is contained in the script, with some supporting info in the yaml. Further data is drawn from a Hilltop server when run (i.e. the data itself is gathered from the server).
NOTE: Hydrobot 0.9.15 does not support all NEMS data sources currently, but more measurements are planned to will be supported in later releases.
Installation (Repeat for each release)
Ensure you have a Python 3.11 interpreter installed (3.12+ is not supported)
In your favourite shell (if you don’t know what that is, use powershell - it’s already installed on windows), create a new virtual environment using this python interpreter and name it “hydrobot0.9.15”. It’s important that this is stored somewhere locally. Assuming it is stored in a “Hydrobot” folder in the C: drive, use the command:
py -3.11 -m venv C:/Hydrobot/hydrobot0.9.15/
Activate this virtual environment. In powershell this should be something like:
C:/Hydrobot/hydrobot0.9.15/Scripts/Activate.ps1
With your venv active, install the latest version of Hydrobot using pip:
pip install hydrobot==0.9.15
Record which version of dependencies you have installed in the location. The following cd changes the location to the relevant point, and the pip freeze records which dependencies are installed by the hydrobot install process for if auditing/reprocessing is required later:
cd C:/Hydrobot/hydrobot0.9.15/ pip freeze > dependencies.txt
Processing Steps
There are 3 main steps for using Hydrobot:
Generate the scripts
Run the scripts
Copy the processed data to your desired destination
If an individual site is being processed, this simply involves creating a copy of the protype scripts *.py and *.yaml for the relevant data source, running the *.py file, and copying data from the resulting xml.
In order to support bulk workflows when doing multiple sites, Hydrobot has the capability to create dsns to make working with the entire processed dataset easier. The process to bulk process data is as follows:
Generate scripts
In your local git repo, navigate to \prototypes\tasks_copy\
Find an appropriate csv or create one if none are appropriate. If the desired sites and data source have “depth”, e.g. lake buoys, they are run through a separate csv to data sources without depth.
All csvs should have “site, from_date, to_date, data_family” as column titles. Site is the site name, from_date and to_date are the dates to process over (can be blank, in which case the archive is used to find the latest processed data for from_date and the current date is used for to_date). Optionally, additional parameters can be set via additional columns (for example, water temperature sets the “standard_measurement_name” via an optional column to deal with the different data sources that water temperature sites have.
If dealing with sites where there are multiple different measurements made at different altitudes (e.g. lake buoys, air temperature), the csv should have a column “depths” which has the different values separated via semicolons (;). This will create a nested folder structure with a batch for each measurement corresponding to each depth.
Open “create_hydrobot_batches.py” (or “create_hydrobot_batches_with_depth.py” if working with depth data sources). Set the parameters “site_list_path”, “destination_path”, and “dsn_destination_path”.
site_list_path is the csv that you wish to generate sites from
destination_path is where your batches will go (where your site folders are)
dsn_destination_path is where dsn(s) will be generated (used for the final step)
Run the create script - this should generate the files. The base files will be drawn from hydrobot templates, with site name, dates, and any other information provided updated for that specific site.
Run the scripts
Make sure your virtual environment is set up (see initial setup instructions) and activate it. To activate, in your shell type the location of the “Activate.ps1” script in the venv/Scripts folder, e.g.:
C:/Hydrobot/hydrobot0.9.15/Scripts/Activate.ps1
You can ensure it is active by typing gcm python and confirm that your python interpreter (under “Source”) is running from your venv folder. You should also see a (venv) at the start of the command line.
To run many scripts, find the .bat file generated in “dsn_destination_path”. Run this bat file from your shell (or copy the text into the shell). This opens many hydrobot scripts in parallel instances (allowing for faster processing).
To run a single script, run the desired python script in the site folder, e.g.:
python hydrobot_run.py
If all goes well, the processing script will generated “merged.html” showing a diagnostic dash for your site. Use this to identify any issues in the site. If any issues are identified, the script/yaml can be modified as appropriate and the script rerun.
Copy processed data
Use Hilltop to open the dsn generated in the first step. Copy all the data into a temporary empty hts. Ensure all the data you expect is in the hts. (Note - the dsn will show most data, but can have some issues displaying, especially with check data).
When happy the data is copyed correctly, copy from the dsn to the desired archive (e.g. ED provisional automation).
In your local git repo, open the prototypesdsn_copyadd_dsn_audit_trial.py script.
At the bottom of the script, you should see a “if __name__==__main__” section which runs the function “write_to_access_for_dsn”. Set “source_dsn” to the path to the dsn (make sure to use an r”” string to avoid escape character), and destination to the path to the archive (again using r strings). Run the file. This file ensures that the audit trail is preserved.
Credits
This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template. Furthermore, Sam is a real champ with the coding and whatnot. Thanks Sam.
Aww thanks Nic. You also da man <3
1 maintainer
Metadata
1 maintainer
Release files for hydrobot 0.9.15
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| hydrobot-0.9.15.tar.gz | 276.8 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| hydrobot-0.9.15-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 413.6 kB
Release files / hydrobot-0.9.15.tar.gz
| Download URL | hydrobot-0.9.15.tar.gz |
|---|---|
| Size | 276.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b7311a6cc8cff60d5eebed4bf8cb75cb6b33a05c5afe93764995b5169d6a9bef
|
|
BLAKE2b-256 checksum How to use checksums |
0aba1c60167097fcef9711a2d72a77ce93b6275f4a0eec2404ec0ebb393742f3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.3
|
Release files / hydrobot-0.9.15-py3-none-any.whl
| Download URL | hydrobot-0.9.15-py3-none-any.whl |
|---|---|
| Size | 136.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9bab91dc9a8304f71b04cd04cef67b31db9717d80230f161cc7fc04c4d375cc7
|
|
BLAKE2b-256 checksum How to use checksums |
5d00e77a22e6d886a1c5e26cf2164ddd7c0e9afd5f595750a28eb7a6703ee02b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.3
|
Security for hydrobot 0.9.15
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
