Ingest frames into the science archive of an observatory control system
Project description
Ingester Library
A library for adding new science data products to an observatory control system's science archive. The library handles uploading files into AWS S3, as well as adding records to the science archive's database containing the searchable metadata of all available files. It records metrics in an openTSDB about ingestion.
Prerequisites
Optional prerequisites may be skipped for reduced functionality.
- Python >= 3.6
- A running science archive
- Write access to the same S3 bucket that the running science archive is using
- (Optional) A running OpenTSDB for metrics collection
Installation
It is highly recommended that you install and run your python code inside a dedicated python virtual environment.
Add the ocs_ingester
package to your python environment:
(venv) $ pip install ocs_ingester
Configuration
AWS and science archive credentials must be set in order to upload data. Science archive configuration as well as the AWS Bucket can be either passed explicitly or set as environment variables. The rest of the configuration must be set as environment variables.
Environment Variables
Variable | Description | Default | |
---|---|---|---|
Science Archive | API_ROOT |
Science Archive URL | "http://localhost:8000/" |
AUTH_TOKEN |
Science Archive Authentication Token. This token must be associated with an admin user. | empty string | |
AWS | BUCKET |
AWS S3 Bucket Name | ingestertest |
AWS_ACCESS_KEY_ID |
AWS Access Key with write access to the S3 bucket | empty string | |
AWS_SECRET_ACCESS_KEY |
AWS Secret Access Key | empty string | |
AWS_DEFAULT_REGION |
AWS S3 Default Region | empty string | |
S3_ENDPOINT_URL |
Endpoint url for connecting to s3. This can be modified to connect to a local instance of s3. | "http://s3.us-west-2.amazonaws.com" |
|
Metrics | OPENTSDB_HOSTNAME |
OpenTSDB Host to send metrics to | empty string |
OPENTSDB_PYTHON_METRICS_TEST_MODE |
Set to any value to turn off metrics collection | False |
|
INGESTER_PROCESS_NAME |
A tag set with the collected metrics to identify where the metrics are coming from | ingester |
|
SUBMIT_METRICS_ASYNCHRONOUSLY |
Optionally submit metrics asynchronously. This option does not apply when the command line entrypoint is used, in which case metrics are always submitted synchronously. Note that some metrics may be lost when submitted asynchronously. | False |
For Developers
Running the Tests
After cloning this project, from the project root and inside your virtual environment:
(venv) $ pip install -e .[tests]
(venv) $ pytest
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for ocs_ingester-3.0.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0c94eae94c4e388537695f5e5a4cfcda70237a2144f41337722b9ae3f7e424d0 |
|
MD5 | 07667dd0afd0a388ceee6e26bfa52138 |
|
BLAKE2b-256 | c03b86127fdc1b7fb9ed7e174e38ff17758125696f0bb425e84febf5d9ffa9cc |