No project description provided
Project description
echofish-aws-raw-to-zarr-lambda
Setting up the Python Environment
MacOS
- Install pyenv (https://github.com/pyenv/pyenv#set-up-your-shell-environment-for-pyenv)
brew update
brew install pyenv
- In ~/.bashrc add
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
brew install openssl readline sqlite3 xz zlib tcl-tk
- Install pyenv-virtualenv (https://github.com/pyenv/pyenv-virtualenv)
brew install pyenv-virtualenv
- In ~/.bashrc add
eval "$(pyenv virtualenv-init -)"
- Open a new terminal
- Install Python version
env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.9.16
- Optional: set global version of Python
env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv global 3.9.16
- Create virtual env
pyenv virtualenv 3.9.16 echofish-aws-raw-to-zarr-lambda
- Set local version of python (if not done already)
- change directory to root of project
pyenv local echofish-aws-raw-to-zarr-lambda
Other OS
- Get a Mac
- See Above MacOS
Setting up IntelliJ
- Install the IntelliJ Python plugin
- Set up pyenv
- File -> Project Structure or CMD + ;
- SDKs -> + -> Add Python SDK -> Virtual Environment
- Select Existing Environment
- Choose ~/.pyenv/versions/echofish-aws-raw-to-zarr-lambda/bin/python
- Set up Python Facet (not sure if this is required)
- File -> Project Structure or CMD + ;
- Facets -> + -> Python
- Set interpreter
Installing Dependencies
- Add dependencies with versions to requirements.txt
pip install --upgrade pip && pip install -r requirements_dev.txt
Maven
This project can use Apache Maven to easily build this project. However, this is not required, but recommended. Maven can be easily installed by downloading it from the Maven site or using sdkman.
Build with Maven
After setting up your pyenv run:
mvn clean install
DOTENV
export AWS_DEFAULT_PROFILE=echofish
Functional Testing
To run a functional test, developer will need to save the associated environment variables in .env for the lambda to use. The ACCESS_KEY and SECRET_ACCESS_KEY can be generated in IAM, users, rudy-dev, security credentials, access keys, create access key. Copy other values as needed from deployed CloudFormation Template.
export OUTPUT_BUCKET=rudy-dev-echofish-118234403147-echofish-dev-output
export TABLE_NAME=rudy-dev-echofish-EchoFish-File-Info
export OUTPUT_BUCKET_ACCESS_KEY="XXX"
export OUTPUT_BUCKET_SECRET_ACCESS_KEY="YYY"
export TOPIC_ARN="arn:aws:sns:us-west-2:118234403147:rudy-dev-echofish-processing-finished"
Deploying new CodePipeline via Cloudformation
Build project w 'mvn clean install', navigate to the 'target/codepipeline' directory. You will need to create a private ECR repo in us-east-1 w name of the project for builds to push to. Then:
aws --profile mggdev cloudformation create-stack --stack-name build-echofish-aws-raw-to-zarr-lambda-trunk --template-body file://codepipeline.yaml --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND
Elastic Container Registry
Note that the main repository is built and curated in the mgg-dev account in us-west-2. The derived docker image is built and pushed to the echofish us-east-1 ECR. That image is then replicated to the echofish us-west-2 account. You will need to go to the echofish ECR -> Repositories -> [repo] -> Permissions and copy the values from an existing project (for both regions).
In the echofish aws account:
IAM -> Roles -> AWSServiceRoleForECRReplication
To Run Tests
python -m pytest
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for echofish-aws-raw-to-zarr-lambda-1.1.0.dev20230930165153.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4c10636a987131442cd5619b6fc3029464b2c4a141dd536bda322b97ae73da6d |
|
MD5 | 6ccce50f2f8238beac6b9dedd311144c |
|
BLAKE2b-256 | ebc5a5f3fc01d2c846c548b04be3ec780d3dc8422118180f481bf611f374d275 |
Hashes for echofish_aws_raw_to_zarr_lambda-1.1.0.dev20230930165153-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | abf40409e92de3ccb471927903cf6dfcf9555ddf020a3c1b9e74db21850afa67 |
|
MD5 | 29e24492016743d3bc8bc9a03b1e5a75 |
|
BLAKE2b-256 | 47ad97a3b5d8d622f8ca0fec37d0dd37b8502186a78be7d73ba1c10247c9e635 |