steampunk-spotter 1.2.1rc1

Quality scanner for Ansible Playbooks

Steampunk Spotter CLI

Table of Contents

• Introduction
• Installation
• Usage
  • Limitations
  • Authentication
  • Scanning
    • Ansible content
    • Selecting the target project
    • Parsing values
    • Uploading metadata
    • Automated application of suggestions to your code
    • Suppressing check result levels
    • Exporting and importing scan payload
    • Scan configuration
    • Formatting scan result
    • Omitting documentation URLs from the output
  • Setting storage folder
  • Disabling colorized output
  • Setting API endpoint
  • CI/CD integrations
    • GitLab
    • GitHub
    • Others
• Development
  • Running from source
  • Building Docker image
• Acknowledgement

Introduction

Steampunk Spotter provides an Assisted Automation Writing tool that analyzes and offers recommendations for your Ansible Playbooks.

The Steampunk Spotter CLI enables the use from the console with the ability to scan Ansible content such as playbooks, roles, collections, or task files.

Installation

steampunk-spotter requires Python 3 and is available as a steampunk-spotter Python package.

$ pip install steampunk-spotter

We suggest installing the package into a clean Python virtual environment.

Usage

After the CLI is installed, you can explore its commands and options by running spotter --help. The --help/-h optional argument is also available for every command.

Limitations

The current release version of Steampunk Spotter contains the following limitations that also apply to the CLI:

• with the FREE subscription plan, you can perform up to 100 scans/month,
• with the INDIVIDUAL, TEAM or ENTERPRISE subscription plan you can perform an unlimited number of scans,
• for the ENTERPRISE subscription plan, contact us at [e-mail] to discuss your needs.

Authentication

To use CLI, you have to log in with your Steampunk Spotter user account. If you don't have an account, use spotter register command that will direct you to the page where you can create one.

To log in, you should provide your username and password:

• using spotter login command
• via --username/-u and --password/-p global optional arguments;
• by setting SPOTTER_USERNAME and SPOTTER_PASSWORD environment variables.

For example:

$ spotter --username <username> --password <password> scan playbook.yaml

or:

$ export SPOTTER_USERNAME=<username>
$ export SPOTTER_PASSWORD=<password>
$ spotter scan playbook.yaml

After that, you can start scanning right away.

Scanning

The CLI spotter scan command is used for scanning Ansible content (playbooks, roles, collections, or task files) and returning back the scan results.

Ansible content

The scan command accepts a positional argument that can be one or many paths to files or directories. The CLI will automatically detect the type of your Ansible content and scan it.

The following types of Ansible content file are currently supported:

• tasks
• playbooks
• roles (applies to tasks and handlers folders)
• collections (applies to roles, playbooks and tests/integration/targets/ folders and any playbooks at the root of collection)

# scan task file, which contains the `tasks` section of the playbook
$ spotter scan path/to/taskfile1.yaml

# scan two playbooks
$ spotter scan path/to/playbook1.yaml path/to/playbook2.yaml

# scan multiple playbooks using glob
$ spotter scan path/to/playbook/folder/play_*.yaml

# scan two roles (scans tasks and handlers folders)
$ spotter scan path/to/role1 path/to/role2

# scan collection (scans Ansible content within roles and playbooks folders
  and playbooks at the root of collection directory)
$ spotter scan path/to/collection

# scan multiple files at once
$ spotter scan path/to/taskfile.yaml path/to/playbook.yaml 
          path/to/role path/to/collection

# scan any folder that contains Ansible content
$ spotter scan path/to/folder

Let us assume we have the following Ansible playbook playbook.yaml file:

---
 1 - name: Sample playbook
 2   hosts: localhost
 3
 4   collections:
 5     - sensu.sensu_go
 6     - ansible.builtin
 7
 8   tasks:
 9     - name: Create Sensu Go user
10       user:
11         name: joe
12         password: "{{ lookup('env', 'SENSU_USER_PASSWORD') }}"
13
14     - name: Read-only command
15       ansible.builtin.command: ls /tmp
16       changed_when: false

In this case, the CLI tool will report something like that back:

$ spotter scan playbook.yaml
playbook.yaml:10:7: ERROR: Potential name clashes for user among
  sensu.sensu_go.user, ansible.builtin.user.
playbook.yaml:10:7: ERROR: Use a fully-qualified name, such as
  sensu.sensu_go.user instead of user.
playbook.yaml:10:7: HINT: Required collection sensu.sensu_go is missing from
  requirements.yml or requirements.yml is missing.
playbook.yaml:15:7: WARNING: Default value for warn parameter changed between
  module versions 2.10.17 and 2.11.0 from true to false. Consider setting
  value of the parameter explicitly.
------------------------------------------------------------------------
Spotter took 0.190 s to scan your input.
It resulted in 2 error(s), 1 warning(s) and 1 hint(s).
Overall status: ERROR

Selecting the target project

This part is only relevant for users with a TEAM plan or higher.

By default, the scan results are stored in the first project of the user's first organization (in the app).

Users that have multiple organizations or projects in the app can use --project-id optional argument to specify the UUID of an existing target project, where the scan result will be stored.

$ spotter scan --project-id <project-id> .

You can learn your project id by logging into the app, selecting the appropriate organization and navigating to the project's dashboard.

Uploading values

By default, CLI parses Ansible YAML content without any parameter values. With values, we can discover additional tips for improvements, so if you want to parse and send the values, you can use --upload-values optional argument.

$ spotter scan --upload-values playbook.yaml

Uploading metadata

By default, CLI collects and uses metadata (file names, line and column numbers, YAML markers) from Ansible content just for displaying the scan output, which means that no data about your Ansible content structure is sent to the backend server. For enriched user experience in the app and to get additional tips for improvements, you can use --upload-metadata optional argument to send the metadata.

$ spotter scan --upload-metadata playbook.yaml

Automated application of suggestions to your code

There is also a --rewrite optional argument that rewrites your files with fixes after scanning. This action will modify your files.

$ spotter scan --rewrite playbook.yaml

Suppressing check result levels

You can use --display-level optional argument for suppressing check result levels. For example, to show only errors (suppress warnings and hints):

$ spotter scan --display-level error playbook.yaml 

Exporting and importing scan payload

To see what data is collected from your Ansible content and sent to the backend server, you can use the optional --export-payload argument.

$ spotter scan --export-payload payload.json playbook.yaml 
Scan data saved to payload.json.
Note: this operation is fully offline. No actual scan was executed.

After that you can also import (with --import-payload optional argument) the exported payload and scan it:

$ spotter scan --import-payload payload.json

Scan configuration

Before scanning, it is possible to configure the environment via the configuration file or optional CLI variables. By default, the CLI runs local discovery of the user's environment.

Currently, the supported variables for configuration file and options are:

• ansible_version - sets target Ansible version

When --config optional argument is used, the variables specified in JSON/YAML config file will overwrite the ones from local discovery. For instance, if we want to overwrite the target Ansible version, we can use the following JSON config file:

{
  "ansible_version": "2.9"
}

And after that we can run the scan command:

$ spotter scan --config config.json playbook.yaml

We can also use --option optional argument that will overwrite local discovery and config file. After --option optional argument, the user should provide configuration via key=value pairs. For instance, if we want to set the Ansible version, we can use the following command:

# scan Ansible playbooks
$ spotter scan --option ansible_version=2.9 playbook.yaml

Formatting scan result

By default, the CLI will output the scan result in plain text format. The --format optional argument allows you to specify the alternative output format of the scan result such as JSON or YAML.

# scan Ansible playbooks
$ spotter scan --format json playbook.yaml

Omitting documentation URLs from the output

In the scan result, the CLI will display a URL to the relevant Ansible content documentation in whenever possible. To omit these documentation URLs from all the output, use --no-docs-url optional argument.

$ spotter scan --no-docs-url playbook.yaml

Setting storage folder

The CLI uses local storage for caching access tokens for the Steampunk Spotter API. The default location is ~/.config/steampunk-spotter, but if you want to change it you can use --storage-path optional argument.

$ spotter --storage-path /my/project/.storage scan playbook.yaml

Disabling colorized output

The CLI will colorize the scan result by default. To make the output non-colorized use --no-colors option.

$ spotter --no-colors scan playbook.yaml

Setting API endpoint

The CLI connects to Steampunk Spotter API (backend server) to perform scanning. The default API endpoint is already set, but if you need to change it, you can do this in different ways.

The precedence of the API endpoint configuration is the following, where the first one specified takes effect:

1. the SPOTTER_ENDPOINT environment variable.

   $ export SPOTTER_ENDPOINT=<spotter-api-url>
   

2. configuration file from storage folder (~/.config/steampunk-spotter/spotter.json) with the root JSON entry "endpoint": "<endpoint>" (this allows persistent API endpoint configuration). You should create it manually with the following JSON content:

   {
     "endpoint": "<endpoint>"
   }
   

3. the default Spotter SaaS API endpoint: https://api.spotter.steampunk.si/api.

CI/CD integrations

The CLI can be used in CI/CD pipelines to set up quality scanning of your Ansible content.

When using the CLI in CI/CD workflows, it is important that you provide Steampunk Spotter credentials as secrets (i.e., pipeline-protected and masked variables).

GitLab

The CLI can be integrated with GitLab CI/CD to display scan results as GitLab’s unit test reports. This means that you will be quickly able to see which checks have failed within your CI/CD pipeline.

This is done by using Spotter CLI tool directly in the CI/CD configuration and configuring it to output your scan result in JUnit XML format, which allows GitLab to display check results as green check marks for successful checks and red crosses for unsuccessful checks. To do so, you should use spotter scan CLI command along with --junit-xml <path-junit-xml> optional argument that will create JUnit XML report at the specified location.

Below is a .gilab-ci.yaml example containing a CI job for the test stage, where you call the aforementioned CLI command and then upload the created JUnit report format XML file as an artifact to GitLab, which will then display it within your pipeline details page or merge request widget.

stages:
  - test

spotter-scan:
  stage: test
  image:
    name: registry.gitlab.com/xlab-steampunk/steampunk-spotter-client/spotter-cli:1.1.0
    entrypoint: [""]
  variables:
    SPOTTER_USERNAME: $SPOTTER_USERNAME
    SPOTTER_PASSWORD: $SPOTTER_PASSWORD
  script:
    - spotter scan --junit-xml report.xml .
  artifacts:
    when: always
    reports:
      junit: report.xml

GitHub

In your CI/CD pipeline, you can specify the name of the Steampunk Spotter GitHub Action (xlab-steampunk/spotter-action@master) with a tag number as a step within your YAML workflow file.

For example, inside your .github/workflows/ci.yml file, you can do:

name: test
on: push
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout this repository
        uses: actions/checkout@v3

      - name: Run Ansible content scan
        uses: ./
        env:
          SPOTTER_USERNAME: ${{ secrets.SPOTTER_USERNAME }}
          SPOTTER_PASSWORD: ${{ secrets.SPOTTER_PASSWORD }}

For comprehensive usage and more examples refer to Steampunk Spotter Action on GitHub Marketplace and Steampunk Spotter GitHub Action repository.

Others

For other CI/CDs, we currently only support using the steampunk-spotter Python package and setting it up as a regular shell command. You can also use spotter-cli Docker image that is available in our GitLab Registry (use registry.gitlab.com/xlab-steampunk/steampunk-spotter-client/spotter-cli:latest image path and select the appropriate tag). You can use spotter scan CLI command along with --junit-xml <path-junit-xml> optional argument to export scan result in JUnit XML format, which is consumed by some CI tools such as Jenkins or Bamboo.

Development

You will first need to clone this repository.

$ git clone https://gitlab.com/xlab-steampunk/steampunk-spotter-client/spotter-cli
$ cd spotter-cli

Running from source

If you want to run directly from the source, run the following commands:

$ python3 -m venv .venv && . .venv/bin/activate
$ pip install -e .

Building Docker image

Before building the Docker image, copy the wheel file distribution to the root of this repository. Make sure that you have only one *.whl file. You can build the wheel yourself or download one from the latest CI/CD pipeline. After that, proceed with the following commands:

$ docker build -t spotter-cli .

After that, you will be able to run scanning in a Docker container. You can mount your Ansible content to the default /scan working directory. The credentials can be specified as environment variables for the Docker container or later via --username and --password CLI optional arguments.

docker run --rm -it -e SPOTTER_USERNAME=<username> -e SPOTTER_PASSWORD=<password> -v "/path/to/your/playbooks/:/scan" spotter-cli scan .

Acknowledgement

This tool was created by XLAB Steampunk, IT automation specialist and leading expert in building Enterprise Ansible Collections.