pycons3rt3 0.0.30

A python3 library for CONS3RT assets and API calls

pycons3rt3

Python3 integration for CONS3RT

Features

• Logging framework
• Python3 SDK for the CONS3RT API (unofficial)
• cons3rt CLI for running cons3rt commands
• ractl CLI for controlling remote access in CONS3RT
• asset CLI for building asset zip files and importing/updating
• Gather CONS3RT deployment info and properties
• Run Linux commands from python
• Configure networking
• Integrate with AWS and Azure
• Nexus Artifact Repository downloads
• Post to Slack
• Post to Teams

Installation

Install from pip

If you have Python 3.6+ installed, you can run one of the following:

pip3 install pycons3rt3 python3 -m pip install pycons3rt3

Install from source

git clone https://github.com/cons3rt/pycons3rt3
cd pycons3rt3
python3 -m venv venv
source venv/bin/activate
python3 -m pip install build
python3 -m pip install .

Directories

When installed, pycons3rt determines and creates a system directory (pycons3rt_system_home) and a local user directory (pycons3rt_user_home).

The default locations for pycons3rt_system_home are here:

• Linux: ~/.cons3rt
• MacOS: ~/.cons3rt
• Windows: C:\cons3rt

The default locations for pycons3rt_user_home is: ~/.cons3rt on all OS types.

PYCONS3RT_HOME environment variable

To change the location of pycons3rt_system_home and pycons3rt_user_home, set the environment variable
PYCONS3RT_HOME to an existing directory. This directory will be used for both system and user home directories.

To do this on Linux or macOS, add the following to ~/.bash_profile:

export PYCONS3RT_HOME="/path/to/desired/directory"

On Windows, add the PYCONS3RT_HOME environment variable to the system settings.

The following directories are also created:

• Config: pycons3rt_system_home/conf/
• Logs: pycons3rt_user_home/log/
• Source: pycons3rt_user_home/src/

If installed from source via asset, the asset clones the pycons3rt source code here for installation:

• pycons3rt_user_home/src/pycons3rt

The logging configuration file is installed here:

• pycons3rt_system_home/conf/pycons3rt-logging.conf

By default, pycons3rt log files will output here:

• pycons3rt_user_home/log/pycons3rt-info.log
• pycons3rt_user_home/log/pycons3rt-warn.log
• pycons3rt_user_home/log/pycons3rt-debug.log

CONS3RT ReST API and CLI

pycons3rt3 provides a python3 SDK for using the CONS3RT ReST API.

To access the ReST API you will need:

• An active CONS3RT Account
• Access to a project
• A CONS3RT project-based ReST API token
• For sites that require Client Certificate Authentication you will need a valid PKI certificate (ECA)

pycons3rt3 does not support CAC authentication at this time

If you have an ECA certificate in p12 or pfx format, convert it to a password-less pem file:

openssl pkcs12 -in certfile.pfx -out certfile.pem -nodes

Configuration

To configure pycons3rt3 for the CONS3RT ReST API type:

cons3rt config

After entering your info, a config file is created here:

pycons3rt_user_home/conf/config.json

This configuration will automatically load for ReST API calls.

asset CLI

The asset CLI command helps you automatically create and import assets:

# Validate an asset structure
asset validate --asset_dir=/path/to/asset

# Create a valid asset zip file
asset create --asset_dir=/path/to/asset

• Creates an asset zip file AssetName.zip in your Downloads directory

Specify the destination directory:

asset create --asset_dir=/path/to/asset --dest_dir=/path/to/directory

• Creates an asset zip file AssetName.zip in the specified directory

Import an asset into CONS3RT using an asset directory:

asset import --asset_dir=/path/to/asset

• Creates an asset zip, and imports the zip file into CONS3RT
• Adds an asset.yml file to the asset directory with the site info and asset ID

Import an asset using a zip file, and keep the zip file:

asset import --zip=/path/to/asset.zip --keep

Import an asset and set the visibility to the project-level:

asset import --asset_dir=/path/to/asset --visibility=OWNING_PROJECT

Update an existing asset from an asset directory using asset_data.yml:

asset update --asset_dir=/path/to/asset --id=12345

Update an existing asset from an asset zip file, specify the asset ID, and keep the zip:

asset update --zip=/path/to/asset.zip --id=12345 --keep

Update an existing asset in CONS3RT by ID:

asset update --asset_dir=/path/to/asset --id=12345

Update an asset using asset data, set log level, but specify which site URL, project, and ID to update:

asset update --asset_dir=/path/to/asset --loglevel=DEBUG --url=api.qa.cons3rt.io --project=jackpine --id=35610

• Uses the asset ID in the asset.yml file
• Creates an asset zip, and updates the asset ID

Update an asset and set the visibility to the community-level:

asset update --asset_dir=/path/to/asset --visibility=COMMUNITY

Query for software assets:

asset query --asset_type=software

Query for the latest community container asset containing "nginx":

asset query --asset_type=containers --asset_subtype=DOCKER --expanded --community --name nginx --latest

Use the queryids command to query for just the latest asset ID for asset with name "nginx":

asset queryids --asset_type=containers --latest --name=nginx

cons3rt CLI

Usage:

cons3rt <command> <subcommands> <args>

Interactive command to create a config.json file:

cons3rt config

This creates a config file at the default location: ~/.cons3rt/conf/config.json

If PYCONS3RT_HOME environment variable is defined, config file location: ${PYCONS3RT_HOME}/conf/config.json

Specify a config file location:

cons3rt --config /path/to/config.json <command> <subcommands> <args>

cons3rt cloud CLI

Permissions:

• Team Manager role

The --id=1 or --ids=1,2,3 args indicate cloud IDs

# Delete a cloud
cons3rt cloud delete --id=1

# List clouds
cons3rt cloud list

# Retrieve cloud details
cons3rt cloud retrieve --id=1

# Share a template from the cloud template provider to all other cloudspaces in the cloud
cons3rt cloud template --id=1 --share --name 'cons3rt-redhat-8'

# Share all templates from the cloud template provider to all other cloudspaces in the cloud
cons3rt cloud template --id=1 --share --all

# List deployment runs in the cloud(s)
cons3rt cloud run list --id=1 
cons3rt cloud run list --ids=1,2,3,4,5

# list deployment run hosts in the cloud(s)
cons3rt cloud run hosts --id=1 
cons3rt cloud run hosts --ids=1,2,3,4,5

# Added CLI command to list GPU usage in the cloud(s)
cons3rt cloud run gpus --id=1
cons3rt cloud run gpus --ids=1,2,3,4,5

# Use the `--load` flag to load deployment run host details saved from prior commands, this works in these commands:
cons3rt cloud run hosts --id=1 --load
cons3rt cloud run gpus --id=1 --load

cons3rt cloudspace CLI

Permissions:

• Team Manager role

The --id=1 or --ids=1,2,3 args can be used to indicate which cloudspace IDs.

• Listing cloudspaces and retrieving cloudspace details

# List cloudspaces
cons3rt cloudspace list

# Retrieve cloudpace details
cons3rt cloudspace retrieve --id=16

• Unregister or deallocate cloudspaces (WARNING: Destructive action)

# Unregister a cloudspace
cons3rt cloudspace unregister --id 123

# Deallocate a cloudspace
cons3rt cloudspace deallocate --id 123

• Cloudspace run actions

# Release active runs from multiple cloudspaces:
cons3rt cloudspace release_active_runs --ids=123,124

# Delete inactive runs from your cloudspace
cons3rt cloudspace delete_inactive_runs --id=123

# List active runs in a cloudspace:
cons3rt cloudspace --list_active_runs --id=123

• Host/power actions on a cloudspace runs

# List runs in cloudspaces
cons3rt cloudspace run list --ids=3,6,9

# Create/Restore/Delete snapshots for all cloudspace runs
cons3rt cloudspace run snapshot create --id=3
cons3rt cloudspace run snapshot restore --id=3
cons3rt cloudspace run snapshot delete --id=3

# Skip run IDs for any of the snapshot commands with --skip
cons3rt cloudspace run snapshot create --id=3 --skip=12345,12346,12347

# Power runs off/on in the cloudspace
cons3rt cloudspace run power on --id=3
cons3rt cloudspace run power off --id=3
cons3rt cloudspace run power restart --id=3

# Delete inactive runs in cloudspaces (does not release active runs)
cons3rt cloudspace run delete --ids=3,4,5

# Release runs in cloudspaces, note this will prompt for confirmation due to the destructive nature
cons3rt cloudspace run release --id=3

• Cloudspace OS template actions

# Delete a specific template
cons3rt cloudspace template delete --id=1 --name 'cons3rt-redhat-8'

# Delete all templates in a cloudspace
cons3rt cloudspace template delete --id=1 --all

# List templates in a cloudspace
cons3rt cloudspace template list --id=1

# Register template in a cloudspace
cons3rt cloudspace template register --id=1 --name 'cons3rt-redhat-8'

# Register multiple templates
cons3rt cloudspace template register --id=1 --names 'cons3rt-redhat-8,cons3rt-redhat-9'

# Register all unregistered templates
cons3rt cloudspace template register --all

# Share a specific template from one cloudspace to another
cons3rt cloudspace template share --provider_id=1 --ids=2,3,4,5,6 --name 'cons3rt-redhat-8'

# Share multiple templates from one cloudspace to another
cons3rt cloudspace template share --provider_id=1 --ids=2,3,4,5,6 --names 'cons3rt-redhat-8,cons3rt-redhat-9'

# Share all templates from one cloudspace to another and specify the max CPU and RAM in Megabytes
cons3rt cloudspace template share --provider_id=1 --ids=2,3,4,5,6 --all --cpu 16 --ram 32000 

cons3rt user CLI

Permissions:

• Site Admin role

# Get a list of active users
cons3rt user list --state=ACTIVE

cons3rt team CLI

Permissions:

• Team Manager role

The --id=1 or --ids=1,2,3 args can be used to indicate which team IDs.

# Get a list of teams (site admins only)
cons3rt team list

# Get a list of team services
cons3rt team service list --id=2

# Get a list of team service users for a particular tool
cons3rt team service users --id=2 --tool=AtlassianJira

# Get a list of all team service users for a team
cons3rt team service users --id=2

# Get a count of ACTIVE users using various collab tools like Jira
cons3rt team collabtools users --id=2

# Get a unique list of ACTIVE users in collab tools projects for a team or list of teams
cons3rt team collabtools users --ids=2,5 --unique

# Get a unique list of ACTIVE + BLOCKED users in collab tools projects for a team or list of teams
cons3rt team collabtools users --ids=2,5 --unique --blocked

# Get a list of active team members, printed by project
cons3rt team members list --id=11

# Get a list of active and blocked team members, printed by project
cons3rt team members list --id=11 --blocked

# Get a unique list of team members for a list of teams
cons3rt team members list --ids=1,2,3,4,5 --unique

# Get a unique list of team members managed by a particular user
cons3rt team members list --username=USERNAME --unique

# Get a list of active team managers
cons3rt team managers

# Get a list of team managers for specific teams
cons3rt team managers --ids=2,5

# Get a list of teams managed by a specific user
cons3rt team managers --username=johndoe

# Get a CSV report of runs in the team
cons3rt team report runs --id=3

# Get a CSV asset report for the team
cons3rt team report assets --id=3

# List runs in team-owned projects
cons3rt team run list --id=3

# Create snapshots for all team runs
cons3rt team run snapshot create --id=3

# Restore snapshots for all team runs
cons3rt team run snapshot restore --id=3

# Delete snapshots for all team runs
cons3rt team run snapshot delete --id=3

# Skip run IDs for any of the snapshot commands with --skip
cons3rt team run snapshot create --id=3 --skip=12345,12346,12347

# Delete inactive runs in a team
cons3rt team run delete --id=3

# Release active runs in a team
cons3rt team run release --id=3

# Power off all runs in a team
cons3rt team run power off --id=3

# Power on all runs in a team
cons3rt team run power on --id=3

# Restart all runs in a team
cons3rt team run power restart --id=3

# Unlock and power off all runs in a team
cons3rt team run power off --id=3 --unlock

# List a team's managed cloudspaces
cons3rt team cloudspace list

cons3rt project CLI

Permissions:

• Team Manager role
• Project Owner or Project Manager role

The --id=1 or --ids=1,2,3 args can be used to indicate which project IDs.

# List projects
cons3rt project list

# Print project details
cons3rt project get --id=3

# List project members
cons3rt project members list --id=3

# List runs in projects
cons3rt project run list --ids=3,6,9

# Create snapshots for all project runs
cons3rt project run snapshot create --id=3

# Restore snapshots for all project runs
cons3rt project run snapshot restore --id=3

# Delete snapshots for all project runs
cons3rt project run snapshot delete --id=3

# Skip run IDs for any of the snapshot commands with --skip
cons3rt project run snapshot create --id=3 --skip=12345,12346,12347

# Power runs off/on in the project
cons3rt project run power on --id=3
cons3rt project run power off --id=3
cons3rt project run power restart --id=3

# Delete inactive runs in projects (does not release active runs)
cons3rt project run delete --ids=3,4,5

# Release runs in projects, note this will prompt for confirmation due to the destructive nature
cons3rt project run release --id=3

cons3rt run CLI

Permissions:

• Asset Developer project role
• Run owner

The --id=1 or --ids=1,2,3 args can be used to indicate which project IDs.

# Create/restore/delete snapshots for all hosts in run
cons3rt run snapshot create --id=3
cons3rt run snapshot restore --id=3
cons3rt run snapshot delete --id=3

# Power runs off/on
cons3rt run power on --id=3
cons3rt run power off --id=3
cons3rt run power restart --id=3

# Delete inactive runs (does not release active runs)
cons3rt run delete --ids=3,4,5

# Release/cancel runs, note this will prompt for confirmation due to the destructive nature
cons3rt run release --ids=3,4,5
cons3rt run cancel --ids=3,4,5

# Download RDP file for a host
cons3rt run rdp --id=3 --host=5

cons3rt host CLI

Permissions:

• Asset Developer project role
• Run owner

The --id=1 or --ids=1,2,3 args can be used to indicate which project IDs.

# Create/restore/delete snapshots a hosts in a run
cons3rt host snapshot create --id=3 --host=5
cons3rt host snapshot restore --id=3 --host=5
cons3rt host snapshot delete --id=3 --host=5

# Power off/on a hosts in a run
cons3rt host power on --id=3 --host=5
cons3rt host power off --id=3 --host=5
cons3rt host power restart --id=3 --host=5

# Download RDP file for a host
cons3rt host rdp --id=3 --host=5

# Resize a host CPU and/or RAM (MB)
cons3rt host resize --id=3 --host=5 --cpu=4 --ram=8192

ractl command -- Controls and queries remote access across the CONS3RT site

Permissions:

• Site Admin role

# Create a csv file with remote access info for a specific cloudspace
ractl print cloudspace --id=116

# Create a csv file with remote access info for the CONS3RT site
ractl print site

# Disable remote access in a cloudspace
ractl cloudspace disable --id=3

# Enable remote access in multiple cloudspaces
ractl cloudspace enable --ids=3,4,5

# Disable/Enable remote access in all cloudspaces under a cloud ID
ractl cloud disable --id=1

# Toggle (disable then enable) remote access in a cloud/cloudspace/site
ractl cloud toggle --ids=1,2,3

• Flags available for enable, disable, and toggle

# Use the `--unlock` flag to force unlock any locked remote access runs
ractl cloud disable --ids=3,4,5 --unlock

# Use the `--load` flag to load remote access data from a previous attempt (time saver)
ractl site toggle --load

# Use the `--skip` flag to skip cloudspaces in a cloud or site action
ractl cloud disable --id=3 --skip=100,101

# Use the `--delay` flag to override the default delay in seconds between each action
ractl cloud toggle --delay=60

• Configure remote access settings on enable or toggle

# Use the `--access` flag to specify an access point override when enabling or toggling remote
# access for a cloudspace (default is the cons3rt-net boundary IP)
ractl cloudspace enable --id=123 --access=50.252.5.6

# Use the `--nordp` flag to disable RDP proxy in the cloudspace remote access configuration
ractl cloudspace enable --id=123 --nordp

# Use the `--port` flag to set the remote access external port in the remote access config
# Default is 9443
ractl cloudspace enable --id=123 --port=443

• Send notifications to Slack or Teams

ractl site toggle --slackurl '<webhook URL>' --slackchannel '#channelName'
ractl site toggle --teamshook '<webhook URL>'

Use pycons3rt3 in python

Logify

With the default configuration log files go to: pycons3rt_user_home/log/, and INFO level is printed to stdout. To customize pycons3rt logging, modify the logging configuration file.

Logging example:

import logging
from pycons3rt3.logify import Logify

mod_logger = Logify.get_name() + '.your_module'
    
    # Then use in a function or class:
    
    class MyClass(object):
        def __init__(self, dep=None):
            self.cls_logger = mod_logger + '.MyCLass'
    	def class_method(self):
    		log = logging.getLogger(self.cls_logger + '.class_method')
    		log.info('Class Method Logging')
    
    def main():
        log = logging.getLogger(mod_logger + '.main')
        log.debug('DEBUG')
    	log.info('INFO')
    	log.warn('WARN')
    	log.error('ERROR')

Deployment

This module provides a set of useful utilities for accessing deployment related info on deployment run hosts.

from pycons3rt3.deployment import Deployment
    
# Create a new Deployment object
dep = new Deployment()
    
# Deployment name
print(dep.deployment_name)
    
# Get the role name
print(dep.cons3rt_role_name)
    
# Deployment properties
print(dep.properties)
    
# Get a specific deployment property value by name
my_value = dep.get_value('cons3rt.user')
    
# Scenario network info
print(dep.scenario_network_info)

# ASSET_DIR
print(dep.asset_dir)

Slack

This module provides an interface for posting anything to Slack!

from pycons3rt3.slack import SlackMessage
from pycons3rt3.slack import SlackAttachments

# Create a message
slack_msg = SlackMessage(
                my_webhook_url, 
                channel='#DevOps', 
                icon_url='http://cool-icon-url',
                text='This is a Slack message',
                user='@sender_username')

# Create and add an attachment
slack_attachment = SlackAttachment(
                       fallback='This is the fallback text', 
                       color='green', 
                       pretext='Pretext', 
                       text='Moar text!')

slack_msg.add_attachment(slack_attachment)

# Send a message
slack_msg.send()

Teams

This module provides an interface for posting anything to Teams! (sorry)

Teams can take a webhook_url arg or will look for the TEAMS_HOOK environment variable.

• Create and send an adaptive card

from pycons3rt3.teams import TeamsMessage

# Create a message
t = TeamsMessage()

# Add a heading
t.add_heading_block_to_card(heading_text='HEADING', color='ok')

# Add a text block to the card
t.add_text_block_to_card(message_text='Message to send')

# Add text blocks for various log levels
t.add_warning_text_block_to_card('WARNING')
t.add_danger_text_block_to_card('DANGER')
t.add_good_text_block_to_card('OK')
t.add_info_text_block_to_card('INFO')

# Add a code block
code_block = """
print('Hello World!')
"""
t.add_code_block_to_card(code_snippet=code_block)

# Send the adaptive card
t.send()

• Use the teams CLI command to send messages

export TEAMS_HOOK=<my webhook URL>

# Send a basic message
teams --text='Text'

# Specify the log level
teams --text='Info' --level='info'
teams --text='Success!' --level='success'
teams --text='Warning!!' --level='warning'
teams --text='Error!!!' --level='error'

# Specify a webhook URL
teams --text='Info' --url='https://<webhook URL>'

# Send a simple Adaptive Card
teams --title='My Heading' --text='Body text of the card' --type=card

# Send a Code Block
teams --title='My Heading' --text='Code snippet' --type=code

Nexus

This module provides simple method of fetching artifacts from a nexus repository.

from pycons3rt3 import nexus

nexus.get_artifact_nexus(
    base_url='https://nexus.example.com',
    repository='releases|snapshots|etc',
    group_id='groupId',
    artifact_id='artifactId',
    packaging='zip|jar|etc',
    destination_dir='/path/to/dest/dir',
    version='x.y.z',
    classifier='classifier',
    suppress_status=True|False,
    overwrite=True|False,
    username='your_username',
    password='your_password'
)

search_results = search_nexus_assets(
    base_url='https://nexus.example.com',
    repository='releases|snapshots|etc',
    group='groupId',
    name='artifactId',
    extension='zip|jar|etc',
    sort_type='version',
    direction='asc|desc',
    version='x.y.z',
    classifier='classifier',
    username='your_username',
    password='your_password'
)

result = search_latest(
    base_url='https://nexus.example.com',
    repository='releases|snapshots|etc',
    group='groupId',
    name='artifactId',
    extension='zip|jar|etc',
    username='your_username',
    password='your_password'
)

nexus cli

The nexus CLI command is an easy way to search for and retrieve artifacts from Nexus v3 using its rest API.

Authentication options:

• Provide both --username and --password args in the CLI call
• Configure a ~/.netrc file, and specify the --netrc arg in the CLI call

Specify a version or latest:

• To get a specific version, use --version x.y.z
• To get the latest, use --latest

For targeting, use: --group, --artifactId, --repo, --classifier, --packaging.

Additional options: --overwrite, --suppress

# Search for a latest release
nexus search --group 'com.cons3rt' --artifactId 'cons3rt-pyotto' --packaging zip --repo releases --netrc --url 'https://nexus.jackpinetech.com' --latest

# Get a latest release
nexus get --group 'com.cons3rt' --artifactId 'cons3rt-pyotto' --repo releases --packaging zip --netrc --url 'https://nexus.jackpinetech.com' --latest

# Get a latest snapshot
nexus get --group 'com.cons3rt' --artifactId 'cons3rt-package' --repo snapshots --packaging zip --netrc --url 'https://nexus.jackpinetech.com' --latest

# Get a specific version
nexus get --group 'com.cons3rt' --artifactId 'cons3rt-pyotto' --repo releases --packaging zip --netrc --url 'https://nexus.jackpinetech.com' --version 24.7.0

s3organizer cli

# Sync all objects from one bucket to another
s3organizer sync --bucket 'source-bucket-name' --targetbucket 'target-bucket-name'

# Sync all objects from one bucket to another, organized under a prefix
s3organizer sync --bucket 'source-bucket-name' --targetbucket 'target-bucket-name' --organize 'path/to/objects'

# Re-encrypt all objects in a bucket, use the same bucket name for source and destination
s3organizer sync --bucket 'encrypted-bucket-name' --targetbucket 'encrypted-bucket-name'

# Use --prefix to filter objects in the source bucket matching a prefix
s3organizer sync --bucket 'source-bucket-name' --targetbucket 'target-bucket-name' --prefix 'sync/under/here'

# Use --length to only include files with a file name length of this many chars
s3organizer sync --bucket 'source-bucket-name' --targetbucket 'target-bucket-name' --length 10

# Use --excludelength to exclude files with a file name length of this many chars
s3organizer sync --bucket 'source-bucket-name' --targetbucket 'target-bucket-name' --excludelength 10

# Delete all keys from a bucket
s3organizer delete --bucket 'source-bucket-name'

# Sync AND dlete all keys from a bucket - use same options with the `all` command
s3organizer all --bucket 'source-bucket-name' --targetbucket 'target-bucket-name' --excludelength 10 --organize 'path/to/objects'

Bash (Linux only)

Executes commands on a Linux system. See the source code for specific available commands but the most commonly used run_command is shown below.

run_command(command, timeout_sec=3600.0, output=True)

Parameters

• command: List containing the command and any additional args
• timeout_sec: (optional) Float specifying how long to wait before terminating the command. Default is 3600.0.
• output: (boolean) True collects the output of the command. In some cases suppressing the command output improves stability.

Returns:

• A dictionary containing "code", the numeric exit code from the command, and "output" which captures the stdout/strerrif output was set True. Sample output:

Raises: CommandError when there is a problem running the command.

{
    "code": "0",
    "output": "stdout/stderr from the command"
}

Example Usage:

from pycons3rt3.bash import run_command
from pycons3rt3.bash import CommandError
command = ['ls', '/root']
try:
    result = run_command(command, timeout_sec=60.0)
    code = result['code']
    output = result['output']
except CommandError:
    raise
if code == 0:
    log.info('Successfully executed command {c}'.format(s=command))
else:
    msg = 'There was a problem running command returned code {c} and produced output: {o}'.format(
                    c=code, o=output)
            log.error(msg)
            raise CommandError(msg)

More to come....