aws-marketplace-toolkit 0.1.4

A CLI tool for managing Stratio AWS Marketplace clusters.

Stratio CLI Tool

Stratio CLI Tool is a Python-based command line interface designed for support and operations teams to manage AWS infrastructures for customers acquired through the AWS Marketplace. It enables you to manage customer accounts, deploy and control clusters, interact with bootstrap EC2 instances during cluster installations, access cluster administration via the KEOS container, and view key operational logs.

Note: This tool is intended for managing clusters that are deployed in customer AWS accounts. It supports operations such as starting/stopping clusters, force-uninstalling clusters, and managing the bootstrap instances that are used during the installation process.

---

Table of Contents

• Overview
• Features
• Architecture & Workflow
• Installation
• Configuration
• Usage
  • General Usage
  • Command Examples
    • Customer Management
    • Cluster Management
    • Bootstrap Instance Management
    • KEOS Operations
    • AWS Console Access
    • Log Operations
  • Filtering

---

Overview

Stratio CLI Tool provides a unified interface to manage AWS-based clusters for customers who have purchased cluster services via the AWS Marketplace. It automates tasks such as:

• Loading a YAML configuration file: Similar to how kubectl uses a kubeconfig.
• Customer and Cluster Management: List, filter, and operate on customer records and their respective clusters.
• Cluster Operations: Start, stop, and force uninstall clusters.
• Bootstrap Instance Handling: Interact with the temporary bootstrap EC2 instance used during cluster installations. These bootstraps remain active during installation (or if an installation fails) and are automatically removed when installation is successful.
• KEOS Container Management: Access the KEOS container which includes administration tools (like kubectl and the keos-installer component) for already installed clusters.
• AWS Console Linking: Generate a link to access the AWS console for a specific customer account.
• Log Management: View logs for cluster start/stop operations and uninstalls. (Installation logs are available through the bootstrap.)

---

Features

• Multi-environment Configuration: Load different configuration files based on environment contexts (e.g., dev, prod) or a custom configuration file.
• Filtering & Custom Output: Support for filtering results using command-line options and displaying output in table or JSON format.
• Interactive & Automated Prompts: Confirmation prompts for potentially destructive actions with an option to bypass for automation.
• Real-time Log Streaming: Automatically stream and monitor log events from CloudWatch, with support for switching to new log streams as they appear.
• Modular Command Structure: Organized subcommands for customers, clusters, bootstraps, KEOS operations, AWS-related actions, and logs.

---

Architecture & Workflow

1. Customer & Cluster Management:
   Customers from the AWS Marketplace are managed using this tool. Each customer has one or more clusters deployed in their AWS account. The CLI provides commands to list and filter these entities.

2. Bootstrap Instances:
   During the installation of a cluster, a temporary bootstrap EC2 instance is deployed. The bootstrap remains available if the installation is ongoing or fails, but is removed automatically upon successful installation.

3. KEOS Container:
   Once a cluster is installed, administration is performed via the KEOS container. This container encapsulates necessary management tools including kubectl and keos-installer.

4. AWS Console Access:
   A dedicated command generates an AWS console link for a specific cluster’s AWS account to facilitate direct AWS management tasks.

5. Log Streaming:
   The CLI streams logs from CloudWatch for start/stop operations and uninstall events. It polls for new log streams and switches to newer streams automatically.

---

Installation

Prerequisites

• Python 3.12+ is required.
• Ensure you have pip installed.
• Poetry is recommended for dependency management and packaging.
• AWS credentials and necessary permissions must be configured for each environment.

Steps

1. Clone this Repository:

   git clone https://github.com/Stratio/aws-marketplace-toolkit
   cd aws-marketplace-toolkit
   

2. Create and Activate a Virtual Environment (Optional): While Poetry creates its own virtual environment by default, you can also manage your environment manually. For example, using venv:

   python3 -m venv venv
   source venv/bin/activate
   

3. Install the Project with Poetry: If you haven't already installed Poetry, follow the installation instructions.

   poetry install
   

   This command will:

   • Create a virtual environment (if one isn’t already active)
   • Install all the project dependencies as specified in your pyproject.toml
   • Apply dynamic versioning using the version specified in the VERSION file

4. Using the CLI Once installed, you can run the CLI tool directly using Poetry:

   poetry run marketplace --help
   

   Or

   poetry shell
   marketplace --help
   

   Alternatively, run directly using:

   ./src/stratio/cli/marketplace.py --help
   

---

Deployment

To deploy the Stratio CLI Tool, follow these steps:

Build the Package

1. Ensure all dependencies are installed:

   poetry install
   

2. Build the package:

   poetry build
   

   This will create the distribution files in the dist directory.

Upload the Package

1. Install Twine if not already installed:

   poetry add --dev twine
   

2. Upload the package to PyPI:

   twine upload dist/*
   

   You will be prompted to enter your PyPI credentials.

By following these steps, you can build and upload your package to PyPI, making it available for installation via pip.

Configuration

The CLI tool uses a YAML configuration file to manage its settings. By default, it looks for a configuration file at:

• ~/.marketplace/config_default.yaml

You can override this by:

• Using a specific context:

  marketplace --context prod customers list
  

  This loads the configuration file at ~/.marketplace/config_prod.yaml.

• Providing a custom configuration file:

  marketplace --config /path/to/your/config.yaml clusters list
  

The configuration file should include necessary API credentials, AWS settings, and other operational parameters. Here is a template of the configuration file:

customers:

  table_filter_regex: "^MarketplaceSubscribers.*$"

  session:
    profile:
      name: "seller@dev"

clusters:

  table_filter_regex: "^MarketplaceSubscribers.*$"
  workspaces_bucket_regex: ".*-automations-stratio-artifacts$"

  keos:
    workspaces_local_folder: /tmp
    base_image: qa.int.stratio.com/stratio/keos-installer
    vault_key: Stratio123

  lambdas:
    marketplace_subscribers_prefix: "MarketplaceSubscribers"
    start_stop_stratio_cluster_prefix: "StartStopStratioCluster"
    start_stratio_applications_prefix: "StartStratioApplications"
    remove_stratio_cluster_prefix: "RemoveStratioCluster"

  root:
    profile:
      name: "root@dev"
  seller:
    profile:
      name: "seller@dev"
  automations:
    profile:
      name: "automations@dev"

Each subcommand also provides its own help text:

marketplace customers --help
marketplace clusters --help
marketplace bootstraps --help
marketplace keos --help
marketplace aws --help
marketplace logs --help

Command Examples

Customer Management

• List all customers:

  marketplace customers list
  

• List customers with filters:

  marketplace customers list --filter "status eq active"
  

Cluster Management

• List clusters (output as JSON):

  marketplace clusters list --output json
  

• Start a cluster (only if in stopped state):

  marketplace clusters start --cluster my-cluster-id
  

• Stop a cluster (automatic confirmation):

  marketplace clusters stop --cluster my-cluster-id --yes
  

• Force uninstall a cluster (only applicable for unsubscribed clusters with a successful installation):

  marketplace clusters uninstall --cluster my-cluster-id
  

Bootstrap Instance Management

• List all bootstrap instances:

  marketplace bootstraps list
  

• Enter a bootstrap instance via SSM session:

  marketplace bootstraps exec --instance i-0123456789abcdef0
  

• Terminate a bootstrap instance:

  marketplace bootstraps terminate --instance i-0123456789abcdef0 --yes
  

KEOS Operations

• Enter a cluster via the KEOS client:

  marketplace keos exec --cluster my-cluster-id
  

• Download the KEOS workspace:

  marketplace keos download --cluster my-cluster-id
  

AWS Console Access

• Generate an AWS console link for a customer account:

  marketplace aws link --account 123456789012
  

• Generate an AWS console link for stored profile:

  marketplace aws link --profile automations@dev
  

Log Operations

• Stream start/stop logs for a cluster:

  marketplace logs cluster --start-stop --cluster my-cluster-id
  

• Stream uninstall logs for a cluster:

  marketplace logs cluster --uninstall --cluster my-cluster-id
  

• List available log groups:

  marketplace logs groups
  

• List log streams for a specific log group:

  marketplace logs streams --group '/aws/stratio/cluster'
  

• Stream logs for a specific log stream:

  marketplace logs read --group '/aws/stratio/cluster' --stream 'stream-name'
  

Filtering

The Stratio CLI Tool supports filtering of customers, clusters, and other resources using a set of operators. Filters are provided as a command-line option (--filter or -f) and should be specified in the following format:

field operator value

Each filter consists of:

• field: The attribute name to filter on (e.g. status, clusterStatus, customerIdentifier).

• operator: One of the following supported operators:

  • eq: Equal to. Matches when the field is equal to the provided value.
  • neq: Not equal to. Matches when the field is not equal to the provided value.
  • gt: Greater than. Matches when the field is greater than the provided value.
  • lt: Less than. Matches when the field is less than the provided value.
  • gte: Greater than or equal to. Matches when the field is greater than or equal to the provided value.
  • lte: Less than or equal to. Matches when the field is less than or equal to the provided value.
  • contains: Contains. Matches when the field contains the provided value.
  • begins_with: Begins with. Matches when the field starts with the provided value.
  • in: In list. Matches when the field's value is within a list of values. Note: The value for the in operator must be provided as a list in the format [value1,value2,...].

• value: The value to compare against. For boolean values, use true or false. For the in operator, supply a comma-separated list enclosed in square brackets (e.g. [active,inactive]).

Examples

• Filter clusters with status equal to "started":

  marketplace clusters list --filter "clusterStatus eq started"
  

• Filter customers where the customer identifier is not "1234":

  marketplace customers list --filter "customerIdentifier neq 1234"
  

• Filter clusters with a node count greater than or equal to 3:

  marketplace clusters list --filter "nodeCount gte 3"
  

• Filter customers whose status is either "active" or "pending":

  marketplace customers list --filter "status in [active,pending]"
  

Multiple filters can be specified by repeating the --filter option. The CLI combines these conditions using logical AND.

Under the hood, these filters are translated into DynamoDB filter expressions, and when multiple filters are provided, their corresponding expressions are combined using the AND operator. This flexible filtering mechanism enables precise querying of resources based on various attributes.

AWS Finder toolkit

Provides a simple python script to locate every single resource belonging to any particular AWS Account. This is useful to identify resources that are not tagged or are not part of any CloudFormation stack. To use this tool, simply type python3 aws-finder.py --help and follow the instructions. The recommended use-case is:

• Create the desired AWS profile in your local machine using python3 manager.py profiles add --name my-profile --customer-provisioned-account-id 0123456789.
• Use with finder the --profile my-profile option to target your specific account.
• Use with finder the --output-file output.json to save all the identified resources for later use.
• Limit the amount of AWS resources to locate. An example command can be found by typing python3 aws_finder.py --help-resources.

Example for Stratio use-case:

python3 finder.py --profile client-profile --region us-east-1 --output-file output.json \
  --with-loadbalancers \
  --with-cloudformation \
  --with-cloudwatch \
  --with-ec2 \
  --with-eks \
  --with-kms \
  --with-s3

AWS Remove toolkit

Provides a simple script to remove every single resource belonging to any particular AWS Account. This is useful to clean up resources that are not tagged or are not part of any CloudFormation stack. To use this tool, simply type ./aws-remove.py --help and follow the instructions. The recommended use-case is:

• Use the previously saved output file from the AWS Finder toolkit.
• Use the previously saved AWS profile from the Manager toolkit.
• Run the script and let it manage dependencies among AWS resources.

AWS Universe toolkit

Contains several utilities for universe resources such as docker images and helm charts. There are a few features included in repositories.py. To use this tool, simply type python3 repositories.py --help and follow the instructions. The recommended use-case is:

• HELM push supports yaml definitions and self-contained zips with the universe of charts.
• Docker push supports individual docker images and lists with the universe of containers.
• Repositories can be removed filtering by prefix, name and tag.

Let's see some examples:

# HELM

# upload helm definition yaml
python3 repositories.py push-helm --aws-profile=mkplc-auto@mkplc-shared --aws-region=eu-west-1 --aws-org-id=o-88l8k59mzn --aws-root-id=r-8815 --aws-ou-id=ou-8815-xqp1203y --charts-file=./helm_14.4.yml --prefix=helm-14.4

# upload self-contained zip with charts (http://qa.int.stratio.com/repository/paas/kubernetes-universe/charts/kubernetes-universe-charts-14.4.1.zip)
python3 repositories.py push-helm --aws-profile=mkplc-auto@mkplc-shared --aws-region=eu-west-1 --aws-org-id=o-88l8k59mzn --aws-root-id=r-8815 --aws-ou-id=ou-8815-xqp1203y --zip-file=./universe-charts-14.4.zip --prefix=helm-14.4

# DOCKER

# upload list file
python3 repositories.py push-docker --aws-profile=mkplc-auto@mkplc-shared --aws-region=eu-west-1 --aws-org-id=o-88l8k59mzn --aws-root-id=r-8815 --aws-ou-id=ou-8815-xqp1203y --images-file=./universe-images_14.4.lst --prefix=universe-14.4.0

# upload single image
python3 repositories.py push-docker --aws-profile=mkplc-auto@mkplc-shared --aws-region=eu-west-1 --aws-org-id=o-88l8k59mzn --aws-root-id=r-8815 --aws-ou-id=ou-8815-xqp1203y --image=docker.io/curlimages/curl:7.85.0 --prefix=universe-14.4.0

# ECR REPOSITORIES

python3 repositories.py delete-repository --prefix=helm-14.4
python3 repositories.py delete-repository --repository=helm-14.4/local-path-provisioner
python3 repositories.py delete-repository --repository=helm-14.4/local-path-provisioner --tag=1.0.0
python3 repositories.py delete-repository --prefix=helm-14.4 --aws-region-list "eu-west-1,eu-central-1"