ansible-content-parser 0.0.4

The ansible content parser

ansible-content-parser

Overview

ansible-content-parser used for analyze Ansible files, such as playbooks, task files, etc. in a given directory.

It runs ansible-lint internally against a given source directory and updates Ansible files (the --fix option of ansible-lint) and generates the lint-result.json file, which summarizes files found in the directory and lint errors.

Build

Execute the tox command. Installable images are created under the dist directory.

Installation

pip install ansible-content-parser

Execution

ansible-content-parser accepts two positional parameters (source and output) with a few optional parameters.

$ ansible-content-parser --help
usage: ansible-content-parser [-h] [--config-file CONFIG_FILE]
                              [--profile {min,basic,moderate,safety,shared,production}] [--skip-transform]
                              [--skip-ansible-lint] [--no-exclude] [-v] [--source-license SOURCE_LICENSE]
                              [--source-description SOURCE_DESCRIPTION] [--repo-name REPO_NAME] [--repo-url REPO_URL]
                              [--version]
                              source output

Parse Ansible files in the given repository by running ansible-lint and generate a training dataset for Ansible
Lightspeed.

positional arguments:
  source                source, which can be an zip/tar archive, a git URL or a local directory
  output                output directory

options:
  -h, --help            show this help message and exit
  --config-file CONFIG_FILE
                        Specify the configuration file to use for ansible-lint. By default it will look for '.ansible-
                        lint', '.config/ansible-lint.yml', or '.config/ansible-lint.yaml' in the source repository.
  --profile {min,basic,moderate,safety,shared,production}
                        Specify which rules profile to be used for ansible-lint
  --skip-transform      Skip the transform step of ansible-lint. If this option is not specified, ansible-lint is
                        executed with the --fix option and files are transformed according to the rules specified.
  --skip-ansible-lint   Skip the execution of ansible-lint.
  --no-exclude          Do not rerun ansible-lint with excluding files that caused syntax check errors. If one or more
                        syntax check errors were found, execution fails without generating the training dataset.
  -v, --verbose         Explain what is being done
  --source-license SOURCE_LICENSE
                        Specify the license that will be included in the training dataset.
  --source-description SOURCE_DESCRIPTION
                        Specify the description of the source that will be included in the training dataset.
  --repo-name REPO_NAME
                        Specify the repository name that will be included in the training dataset. If it is not
                        specified, it is generated from the source name.
  --repo-url REPO_URL   Specify the repository url that will be included in the training dataset. If it is not
                        specified, it is generated from the source name.
  --version             show program's version number and exit

source positional argument

The first positional parameter is source, which specifies the source repository to be used. Following three types of sources are supported:

1. File directory.
2. Archive file in the following table:

File Format	File Extension
ZIP	.zip
Uncompressed TAR	.tar
Compressed TAR	.tar.gz, .tgz, .tar.bz2, .tbz2, .tar.xz, .txz

3. Git URL, e.g. [e-mail]:ansible/workshop-examples.git or https://github.com/ansible/workshop-examples.git

output positional argument

The second positional parameter is output, which specifies a writable directory. If the directory already exists, it has to be an empty directory. If it does not exist, it will be newly created with the given name.

ansible-content-parser creates therepository subdirectory in the output directory and copies the contents of the source repository to it. The copied contents may be changed by during the execution of the Content Parser.

Outputs

Following directory structure is created in the directory specified with the output positional argument.

output/
  |-- ftdata.json  # Training dataset
  |-- report.txt   # A human-readable report
  |
  |-- repository/
  |     |-- (files copied from the source repository)
  |
  |-- metadata/
        |-- lint-result.json     # Metadata generated by ansible-lint
        |-- sarif.json           # ansible-lint results in SARIF format
        |-- (other metadata files generated)

ftdata.json

This is the training dataset file, which is the main output of ansible-content-parser.

It is in the JSONL format, each of whose line represents a JSON object

report.txt

This is a human-readable report that provides the summary information of the run of ansible-content-parser, which contains sections like:

1. File counts per type
2. List of Ansible files identified
3. Issues found by ansible-lint
4. List of Ansible modules found in tasks

Note: When the --skip-ansible-lint option is specified, the first three sections do not appear in the report.

metadata directory

This subdirectory contains a few files that contain metadata generated in the Content Parser run.

lint-result.json

lint-result.json is created in the metadata subdirectory as the result of the execution of ansible-content-parser. The file contains a dictionary, which has two key/value pairs:

1. files This is for the list of files that were found in the execution. The format of each file entry is explained below.

Each file entry is represented as a dictionary that contains following keys

Key	Description
base_kind	MIME type of the file, for example, text/yaml
dir	Directory where the file resides.
exc	Exception found while processing this file. It can be null.
filename	File name
kind	File type, for example, playbook, tasks or role
name	File name (Usually same as filename)
parent	Name of the parent, like a role. It can be null
role	Ansible role. It can be null
stop_processing	Identifies whether processing was stopped on this file or not
updated	Identifies whether contents were updated by ansible-lint

Following shows an example of a file entry:

{
  "base_kind": "text/yaml",
  "dir": "/mnt/input/roles/delete_compute_node/tasks",
  "exc": null,
  "filename": "roles/delete_compute_node/tasks/main.yaml",
  "kind": "tasks",
  "name": "roles/delete_compute_node/tasks/main.yaml",
  "parent": "roles/delete_compute_node",
  "role": "delete_compute_node",
  "stop_processing": false,
  "updated": false
}

2. excluded This is for the list of file paths, which were excluded in the second ansible-lint execution because syntax check errors were found in those files on the first execution. The files included in the list will not appear in the entries associated with the files key.

• Note: If ansible-content-parser is executed with the --no-exclude option, the second execution does not occur even if syntax check errors were found on the first execution and the training dataset will not be created.

sarif.json

This is the output of ansible-lint with the --sarif-file option. The report.txt contains a summary generated from this file in the "Issues found by ansible-lint" section.