Skip to main content

A command line to assist McCloud users.

Project description

Intro

mccloud_cli is a command line interface to the McCloud service. McCloud provides a simple means to run a Shasta genome assembly in the cloud. mccloud_cli can generate presigned URLs for use with the McCloud web UI, and submit McCloud jobs directly from the command line.

To use mccloud_cli, you must have the AWS command line set up and working correctly with your AWS account. Please refer to the AWS CLI User Guide for more information on this prerequisite.

Quick start

  1. Ensure you have the AWS CLI installed and working correctly with your account. A good test is the ability to successfully move data in and out of your own S3 bucket (e.g., aws --profile your-aws-profile-name s3 ls s3://your-bucket-name/)
  2. [optional] We highly reccommend that you run all python programs in a virtual environment. Set up and activate a virtual environment.
  3. pip install mccloud-cli
  4. mccloud_cli --help

Presigned URLs example

mccloud_cli can generate AWS presigned URLs. This is useful when you want to store your data in your own AWS S3 bucket (i.e., privately), but temporarily grant McCloud access to read input data, and write results back to your bucket.

IMPORTANT: AWS S3 presigned URLs are a standard way to share data, but make the assumption that you won't share the URL with anyone. For more information on presigned URLs, please read the AWS S3 User Guide.

Usage:

$ mccloud_cli generate-s3-presigned-url --help
Usage: mccloud_cli generate-s3-presigned-url [OPTIONS]

Options:
  --profile TEXT             Use a specific profile from your AWS credential file.
  --no-confirmation BOOLEAN  Perform requested actions without user confirmation.
  -i, --input-url TEXT       S3 URL which will be presigned to allow READ (input) access.
  --expires-in INTEGER       Number of seconds until the pre-signed URLs expire.  [default: 172800]
  --help                     Show this message and exit.

For example:

$ mccloud_cli generate-s3-presigned-url --profile my-aws-profile -i s3://my-bucket/reads.fasta.gz -o s3://my-bucket/results.tar.gz

This action will generate AWS S3 presigned URLs for the input and output URLs you have specified, with an expiration time 172800 seconds [2 days] from now.

Read (input) URLs:
------------------
s3://my-bucket/reads.fasta.gz

Do you wish to proceed? [y/N]: y

Your presigned URLs will expire in 24 hours.

Please launch the McCloud job with these URLs.

Presigned INPUT URLs:
---------------------
https://s3.amazonaws.com/my-bucket/reads.fasta.gz?AWSAccessKeyId=...

Job submission example

mccloud_cli will also submit jobs directly from the command line. Input and output URLs must be https URLs or s3 URLs in a bucket you have access to.

Usage:

$ mccloud_cli submit-job --help
Usage: mccloud_cli submit-job [OPTIONS]

Options:
  --mcat TEXT                     McCloud Access Token, ie, MCAT-****.  [required]
  --email EMAIL ADDRESS           Email address for job notifications.  [required]
  -i, --input-url TEXT            S3 URL which will be presigned to allow READ (input) access.  [required]
  -o, --output-url TEXT           A single S3 URL which McCloud has write access to.  [required]
  --shasta-version TEXT           [required]
  --shasta-config TEXT            [required]
  --shasta-cli-opts TEXT
  --I-have-read-and-agree-to-McCloud-terms-of-use
                                  Confirm that you have read, and agree to, the McCloud terms of use, available at
                                  https://mccloud.czi.technology/terms-of-use
  --tail-job-log                  After job is submitted, print out job log messages.
  --profile TEXT                  Use a specific profile from your AWS credential file.
  --help                          Show this message and exit.

For example:

$ mccloud_cli submit-job --mcat 'MCAT-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX' --email my@email.com --profile my-aws-profile -i s3://my-bucket/reads.fasta.gz -o s3://my-bucket/output-folder --shasta-version 0.7.0 --shasta-config Nanopore-Sep2020
I have read and agree to mccloud terms of use [y/N]: Y
Starting McCloud job, id = ec39baa4-ea68-4c1e-8040-40277b2a48a0
You can view logs at https://mccloud.czi.technology/jobs/ec39baa4-ea68-4c1e-8040-40277b2a48a0/monitor

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mccloud_cli-0.0.8.tar.gz (10.3 kB view details)

Uploaded Source

Built Distribution

mccloud_cli-0.0.8-py3-none-any.whl (10.7 kB view details)

Uploaded Python 3

File details

Details for the file mccloud_cli-0.0.8.tar.gz.

File metadata

  • Download URL: mccloud_cli-0.0.8.tar.gz
  • Upload date:
  • Size: 10.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.0

File hashes

Hashes for mccloud_cli-0.0.8.tar.gz
Algorithm Hash digest
SHA256 ec9960b2d85bcc29051ed0cda6ed1d79e5b7eab8e51ef478eb5d2f1c6a0807f3
MD5 4222bd8bebe161a7eafec7e72f906294
BLAKE2b-256 069bb4d6698ee9839f52af01ee4bd9e05d049e2baf097fdcef90a365807eebbe

See more details on using hashes here.

Provenance

File details

Details for the file mccloud_cli-0.0.8-py3-none-any.whl.

File metadata

  • Download URL: mccloud_cli-0.0.8-py3-none-any.whl
  • Upload date:
  • Size: 10.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.0

File hashes

Hashes for mccloud_cli-0.0.8-py3-none-any.whl
Algorithm Hash digest
SHA256 1abe6eda5256a7f568531cbfdc788ba689d6f86def90188f7149e476a65f45e8
MD5 d9c3edcaba013b35d87bb407164f30b7
BLAKE2b-256 29c2c10d40e0baa30722a102abc5f6c2f75285e2ae166c4bada577ab8dc9cef2

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page