Skip to main content

No project description provided

Project description

echofish-aws-raw-to-zarr-lambda

Setting up the Python Environment

MacOS

  1. Install pyenv (https://github.com/pyenv/pyenv#set-up-your-shell-environment-for-pyenv)
    1. brew update
    2. brew install pyenv
    3. In ~/.bashrc add
      1. export PYENV_ROOT="$HOME/.pyenv"
      2. export PATH="$PYENV_ROOT/bin:$PATH"
      3. eval "$(pyenv init -)"
    4. brew install openssl readline sqlite3 xz zlib tcl-tk
  2. Install pyenv-virtualenv (https://github.com/pyenv/pyenv-virtualenv)
    1. brew install pyenv-virtualenv
    2. In ~/.bashrc add
      1. eval "$(pyenv virtualenv-init -)"
  3. Open a new terminal
  4. Install Python version
    1. env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.9.16
    2. Optional: set global version of Python env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv global 3.9.16
  5. Create virtual env
    1. pyenv virtualenv 3.9.16 echofish-aws-raw-to-zarr-lambda
  6. Set local version of python (if not done already)
    1. change directory to root of project
    2. pyenv local echofish-aws-raw-to-zarr-lambda

Other OS

  1. Get a Mac
  2. See Above MacOS

Setting up IntelliJ

  1. Install the IntelliJ Python plugin
  2. Set up pyenv
    1. File -> Project Structure or CMD + ;
    2. SDKs -> + -> Add Python SDK -> Virtual Environment
    3. Select Existing Environment
    4. Choose ~/.pyenv/versions/echofish-aws-raw-to-zarr-lambda/bin/python
  3. Set up Python Facet (not sure if this is required)
    1. File -> Project Structure or CMD + ;
    2. Facets -> + -> Python
    3. Set interpreter

Installing Dependencies

  1. Add dependencies with versions to requirements.txt
  2. pip install --upgrade pip && pip install -r requirements_dev.txt

Maven

This project can use Apache Maven to easily build this project. However, this is not required, but recommended. Maven can be easily installed by downloading it from the Maven site or using sdkman.

Build with Maven

After setting up your pyenv run: mvn clean install

DOTENV

export AWS_DEFAULT_PROFILE=echofish

Functional Testing

To run a functional test, developer will need to save the associated environment variables in .env for the lambda to use. The ACCESS_KEY and SECRET_ACCESS_KEY can be generated in IAM, users, rudy-dev, security credentials, access keys, create access key. Copy other values as needed from deployed CloudFormation Template.

export OUTPUT_BUCKET=rudy-dev-echofish-118234403147-echofish-dev-output
export TABLE_NAME=rudy-dev-echofish-EchoFish-File-Info
export OUTPUT_BUCKET_ACCESS_KEY="XXX"
export OUTPUT_BUCKET_SECRET_ACCESS_KEY="YYY"
export TOPIC_ARN="arn:aws:sns:us-west-2:118234403147:rudy-dev-echofish-processing-finished"

Deploying new CodePipeline via Cloudformation

Build project w 'mvn clean install', navigate to the 'target/codepipeline' directory. You will need to create a private ECR repo in us-east-1 w name of the project for builds to push to. Then:

aws --profile mggdev cloudformation create-stack --stack-name build-echofish-aws-raw-to-zarr-lambda-trunk --template-body file://codepipeline.yaml --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND

Elastic Container Registry

Note that the main repository is built and curated in the mgg-dev account in us-west-2. The derived docker image is built and pushed to the echofish us-east-1 ECR. That image is then replicated to the echofish us-west-2 account. You will need to go to the echofish ECR -> Repositories -> [repo] -> Permissions and copy the values from an existing project (for both regions).

In the echofish aws account:

IAM -> Roles -> AWSServiceRoleForECRReplication

To Run Tests

python -m pytest

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

Built Distribution

File details

Details for the file echofish-aws-raw-to-zarr-lambda-1.1.0.dev20230930183050.tar.gz.

File metadata

File hashes

Hashes for echofish-aws-raw-to-zarr-lambda-1.1.0.dev20230930183050.tar.gz
Algorithm Hash digest
SHA256 5b510b7455fcf8bdf56b3921341b2b8374f09b4fbce0ecfe8c1d69ffa0b2e5a3
MD5 c6be41ba161d6abddb5aecb0f8790eeb
BLAKE2b-256 88f68aa25de1780604826cea8316dfd860b721a9b33aa3ba1a07bf7d781fdb4a

See more details on using hashes here.

File details

Details for the file echofish_aws_raw_to_zarr_lambda-1.1.0.dev20230930183050-py3-none-any.whl.

File metadata

File hashes

Hashes for echofish_aws_raw_to_zarr_lambda-1.1.0.dev20230930183050-py3-none-any.whl
Algorithm Hash digest
SHA256 58a7be71227b8b09adcf3425f067e2b513afe472d24892d422b9dd6e328e42ce
MD5 42058c596ecce27e2ca87d9bdea77b68
BLAKE2b-256 1ab17c90e5ea765dac64efba7456f82e94f580249dc55177664ac3c6d2d265c5

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page