New process
Steps
- Download xml
- Create basic json
- Upload json to s3
- munge basic json to parquet
- munge parquet to
- experiment joined
- sample joined
- run joined
- study with aggregates
- Include aggs in spark jobs:
- number of samples, experiments, runs
- sample, experiment, and run accessions (as array)
- Save munged spark data (json, parquet)
- Create elasticsearch index mappings
- Drop existing elasticsearch mappings
- Load elasticsearch index mappings
lambda
zip lambdas.zip lambda_handlers.py sra_parsers.py
aws lambda create-function --function-name sra_to_json --zip-file fileb://lambdas.zip --handler lambda_handlers.lambda_return_full_experiment_json --runtime python3.6 --role arn:aws:iam::377200973048:role/lambda_s3_exec_role
Invoke
aws lambda invoke --function-name sra_to_json --log-type Tail --payload '{"accession":"SRX000273"}' /tmp/abc.txt
Concurrency
1000 total, reserve for certain functions to limit, etc.
aws lambda put-function-concurrency --function-name sra_to_json --reserved-concurrent-executions 20
timeout and memory
aws lambda update-function-configuration --function-name sra_to_json --timeout 15
logging
https://github.com/jorgebastida/awslogs
awslogs get /aws/lambda/sra_to_json ALL --watch
dynamodb
aws dynamodb scan --table-name sra_experiment --select "COUNT"
GEO
python -m omicidx.geometa --gse=GSE10
Will print json, one "line" per entity to stdout.
1 maintainer
Release files for omicidx 0.3.8
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| omicidx-0.3.8.tar.gz | 34.5 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| omicidx-0.3.8-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 76.6 kB
Release files / omicidx-0.3.8.tar.gz
| Download URL | omicidx-0.3.8.tar.gz |
|---|---|
| Size | 34.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0b8b503a958c5da686e7694f9fc460083955a3bf3de639ca12f84f59bf4ab7ac
|
|
BLAKE2b-256 checksum How to use checksums |
12bf536361131fd58081a3c930d2d464934b5d2087a195f23349a44fc09061cc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/0.12.17 CPython/3.7.3 Darwin/18.2.0
|
Release files / omicidx-0.3.8-py3-none-any.whl
| Download URL | omicidx-0.3.8-py3-none-any.whl |
|---|---|
| Size | 42.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e8e10230788eed839263e3015c61a5d165f3bef699f5a41b98c36dc1794cd62b
|
|
BLAKE2b-256 checksum How to use checksums |
a37e925cccd8b25b9122e6002d44f92e25c937faed28baa5d5482663b97db55e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/0.12.17 CPython/3.7.3 Darwin/18.2.0
|
