Skip to main content

Ingest frames into the LCO Archive

Project description

Ingester Library

Build Status Codacy Badge

A library for adding new science data products to an observatory control system's science archive. The library handles uploading FITS files into AWS S3, as well as adding records to the science archive's database containing the searchable metadata of all available FITS files.

Prerequisites

Optional prerequisites may be skipped for reduced functionality.

  • Python >= 3.6
  • A running science archive
  • Write access to the same S3 bucket that the running science archive is using
  • (Optional) A running OpenTSDB for metrics collection

Installation

It is highly recommended that you install and run your python code inside a dedicated python virtual environment.

Add the lco_ingester package to your python environment:

(venv) $ pip install lco_ingester

Configuration

AWS and science archive credentials must be set in order to upload data. Science archive configuration as well as the AWS Bucket can be either passed explicitly or set as environment variables. The rest of the configuration must be set as environment variables.

Environment Variables

Variable Description Default
Science Archive API_ROOT Science Archive URL "http://localhost:8000/"
AUTH_TOKEN Science Archive Authentication Token. This token must be associated with an admin user. empty string
AWS BUCKET AWS S3 Bucket Name ingestertest
AWS_ACCESS_KEY_ID AWS Access Key with write access to the S3 bucket empty string
AWS_SECRET_ACCESS_KEY AWS Secret Access Key empty string
AWS_DEFAULT_REGION AWS S3 Default Region empty string
Metrics OPENTSDB_HOSTNAME OpenTSDB Host to send metrics to empty string
OPENTSDB_PYTHON_METRICS_TEST_MODE Set to any value to turn off metrics collection False
INGESTER_PROCESS_NAME A tag set with the collected metrics to identify where the metrics are coming from ingester
SUBMIT_METRICS_ASYNCHRONOUSLY Optionally submit metrics asynchronously. This option does not apply when the command line entrypoint is used, in which case metrics are always submitted synchronously. Note that some metrics may be lost when submitted asynchronously. False
Postprocessing FITS_BROKER FITS exchange broker memory://localhost
PROCESSED_EXCHANGE_NAME Processed files RabbitMQ Exchange Name archived_fits
POSTPROCESS_FILES Optionally submit files to fits queue True

For Developers

Running the Tests

After cloning this project, from the project root and inside your virtual environment:

(venv) $ pip install -r requirements.txt
(venv) $ pytest

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lco-ingester-2.1.15.tar.gz (16.4 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page