Ingester Library

A library for adding new science data products to an observatory control system's science archive. The library handles uploading FITS files into AWS S3, as well as adding records to the science archive's database containing the searchable metadata of all available FITS files.

Prerequisites

Optional prerequisites may be skipped for reduced functionality.

Python >= 3.6
A running science archive
Write access to the same S3 bucket that the running science archive is using
(Optional) A running OpenTSDB for metrics collection

Installation

It is highly recommended that you install and run your python code inside a dedicated python virtual environment.

Add the lco_ingester package to your python environment:

(venv) $ pip install lco_ingester

Configuration

AWS and science archive credentials must be set in order to upload data. Science archive configuration as well as the AWS Bucket can be either passed explicitly or set as environment variables. The rest of the configuration must be set as environment variables.

Environment Variables

	Variable	Description	Default
Science Archive	`API_ROOT`	Science Archive URL	`"http://localhost:8000/"`
	`AUTH_TOKEN`	Science Archive Authentication Token. This token must be associated with an admin user.	empty string
AWS	`BUCKET`	AWS S3 Bucket Name	`ingestertest`
	`AWS_ACCESS_KEY_ID`	AWS Access Key with write access to the S3 bucket	empty string
	`AWS_SECRET_ACCESS_KEY`	AWS Secret Access Key	empty string
	`AWS_DEFAULT_REGION`	AWS S3 Default Region	empty string
Metrics	`OPENTSDB_HOSTNAME`	OpenTSDB Host to send metrics to	empty string
	`OPENTSDB_PYTHON_METRICS_TEST_MODE`	Set to any value to turn off metrics collection	`False`
	`INGESTER_PROCESS_NAME`	A tag set with the collected metrics to identify where the metrics are coming from	`ingester`
	`SUBMIT_METRICS_ASYNCHRONOUSLY`	Optionally submit metrics asynchronously. This option does not apply when the command line entrypoint is used, in which case metrics are always submitted synchronously. Note that some metrics may be lost when submitted asynchronously.	`False`
Postprocessing	`FITS_BROKER`	FITS exchange broker	`memory://localhost`
	`PROCESSED_EXCHANGE_NAME`	Processed files RabbitMQ Exchange Name	`archived_fits`
	`POSTPROCESS_FILES`	Optionally submit files to fits queue	`True`

For Developers

Running the Tests

After cloning this project, from the project root and inside your virtual environment:

(venv) $ pip install -r requirements.txt
(venv) $ pytest

Metadata

Release files for lco-ingester 2.1.15

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for lco-ingester 2.1.15
File	Size	Uploaded
lco-ingester-2.1.15.tar.gz	16.4 kB	Jun 3, 2020	Details

Release files / lco-ingester-2.1.15.tar.gz

Download URL	lco-ingester-2.1.15.tar.gz
Size	16.4 kB
Tags	Source
SHA-256 checksum How to use checksums	`3f1d2c40fea07a49fba3bddb6943945214d8706b3a7bd46a7534fbf7ef574321`
BLAKE2b-256 checksum How to use checksums	`e926747daec313d9cda7f31eacb4750495f34ae9a8382acff58d02e9836ee4f9`
Upload date	Jun 3, 2020
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/47.1.1 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.7.1`