insightconnect-plugin-runtime 6.2.0

InsightConnect Plugin Runtime

InsightConnect Python Plugin Runtime

The InsightConnect Python Plugin Runtime is used for building plugins in Python for Rapid7 InsightConnect. The project results in the building and publishing of two components:

• Python Plugin Runtime Library
• Base InsightConnect Plugin Docker Images

Docker images created during the build and deployment of this project are uploaded to the Rapid7 Docker Repositories.

Further documentation for building an InsightConnect plugin is available to get started.

Development of the InsightConnect Plugin Runtime

The Python Runtime codebase is built to support Python 3.11.10 as of version 6.2.0. The following dependencies will need to be installed when developing or testing the Plugin Runtime:

• Python 3.11.10
• Docker
• make
• tox

Getting Started

Building Python Library

To build and install the plugin runtime library locally, first create a Python virtual environment for the particular Python version and activate it. Then build, install, and confirm the package has been installed.

> python3 -m venv venv
> source venv/bin/activate
> pip install -e ./
> pip list | grep insightconnect-plugin-runtime
insightconnect-plugin-runtime 5.2.2

Building the InsightConnect Plugin Runtime Docker Images

Currently the python-3 dockerfile is used by default when building the docker image. If you want to specify another dockerfile for testing purposes, such as python-3-slim, you can pass it as an argument.

make build-image DOCKERFILE=python-3-slim

This will overwrite the default python-3, provided that it exists in the dockerfiles directory.

Building/troubleshooting local images (Mac M1)

Our images are built with OS/ARCH:linux/amd64 due to our ci.yml file specifying linux. Historically this has not caused any issues but with an M1 Mac, docker will now use arm64. This is because docker builds the image to match the distribution of the OS running the command. To ensure that building locally and on GH produces an image of the same architecture the --platform specifier is passed in the dockerfile.

If you build a plugin locally we can see further issues that:

• the local image does match the current OS distro so docker continues to look remotely..
• docker can not find this (newly created) image remotely on docker registry (dev images should not be pushed to registry)

To overcome this we need to specify platform=linux/amd64 in the plugin docker file. E.g. if building plugins/html then this file should be updated plugins/html/Dockerfile. Note: building this arch on mac M1 will cause the build to be slower.

Example error of local image not matching OS distro without passing the --platform flag:

 => ERROR [internal] load metadata for docker.io/rapid7/insightconnect-python-3-slim-plugin:latest                          1.4s
------
 > [internal] load metadata for docker.io/rapid7/insightconnect-python-3-slim-plugin:latest:
------
Dockerfile:1
--------------------
   1 | >>> FROM rapid7/insightconnect-python-3-slim-plugin
   2 |     LABEL organization=rapid7
   3 |     LABEL sdk=python
--------------------
ERROR: failed to solve: rapid7/insightconnect-python-3-slim-plugin: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
make: *** [image] Error 1

Testing Sample Plugin

The easiest way to test changes to the runtime is by running it locally against one of the sample plugins included in the repository. Make sure a virtual environment has been activated and then pass in the sample directory name as a parameter:

> make sample=example run-sample

The plugin will be started in http mode and listening at http:0.0.0.0:10001:

[2020-02-13 23:21:13 -0500] [56567] [INFO] Starting gunicorn 19.7.1
[2020-02-13 23:21:13 -0500] [56567] [INFO] Listening at: http://0.0.0.0:10001 (56567)
[2020-02-13 23:21:13 -0500] [56567] [INFO] Using worker: threads
[2020-02-13 23:21:13 -0500] [56571] [INFO] Booting worker with pid: 56571

To override Gunicorn config export environment variable GUNICORN_CONFIG_FILE pointing to json file with config See example in samples/example/gcfg.json

export GUNICORN_CONFIG_FILE ="./gcfg.json"

To build, install, and run runtime changes without the use of the run-sample rule, the below steps can be used for same result:

> python setup.py build && python setup.py install
> cd samples/example
> python setup.py build && python setup.py install
> ./bin/icon_example http

Testing Locally with Docker Runtime

In addition to testing locally with the resulting runtime and an InsightConnect plugin, it is also possible to build a plugin locally and test it as it would be used by the InsightConnect orchestrator.

First, build the base runtime:

make build-image

This will result in tagged Docker images that can be used in the included sample plugins. Then the plugin can be built and run locally:

> cd samples/example
> icon-plugin build image --no-pull
> docker run -it -p 10001:10001 rapid7/example:latest http

Running Tests

In order to run tests, first ensure tox has been installed. Tox makes it easy for testing this project in isolated virtual environments and for specific Python versions. To install tox:

> pip install tox

Running all tests:

> tox

Running a specific test file:

> tox -e py39 -- tests/plugin/hello_world/tests/test_cli.py

Plugin vs Slim Plugin Comparison

	Plugin	Slim Plugin
Python Version	3.11.10	3.11.10
OS	Alpine	Bullseye
Package installer	apk	apt
Shell	/bin/sh	/bin/bash
Image Size	~370MB	~230MB

Note that for the plugin image, we run apk update and apk add .. which leads to a longer build time.

Release

To release a new version of the InsightConnect Python Plugin Runtime, the below steps must be followed:

1. Make sure that version is updated in changelog, and setup.py file.
2. Create a Pull Request with your changes to be merged into master
3. Merge changes after receiving at least one approval
4. Create a versioned tag off of release; version must follow SemVer; using the git tag [version #] command (not through the GitHub release UI!) then git push origin --tags.
5. Github Action will perform a matrix build and release based on the recently created tag for each Python version and Dockerfile in scope

Contributions

Contributions for maintaining and enhancing the InsightConnect Python Plugin Runtime are appreciated. This project uses Black for code formatting and includes a pre-commit hook to auto format code as it is contributed. Black is installed as a test dependency and the hook can be initialized by running pre-commit install after cloning this repository.

Changelog

• 6.2.0 - Update base images to pull Python 3.11.10 | changed the pep-8 check in tox to pycodestyle
• 6.1.4 - Address vulnerabilities within local development requirements.txt and vulnerabilities in slim image.
• 6.1.3 - Addressing failing Python Slim package (bump packages).
• 6.1.2 - Removing the call to mask_output_data for task code.
• 6.1.1 - Addressed vulnerabilities within the slim and non-slim Python images (bumping packages).
• 6.1.0 - Update AWSAction tests to utilise AWS Client Error names for connection test | Update request_error_handling to include input for custom configuration
• 6.0.1 - Address vulnerabilities within the following python packages: Jinja2, requests, urllib3, zipp and setuptools.
• 6.0.0 - Address vulnerabilities within certifi python package | Bump the version of OpenSSL used | If running a task connection test, return a custom message from the task test and not the full log.
• 5.6.1 - Making sure all paths that can call the task connection test endpoints return a 400 error if the test fails
• 5.6.0 - Add APIException class for error handling | Fix error in response_handler where data of type Response was not correctly being returned
• 5.5.5 - Address bug with typing for type Dict in Python 3.8
• 5.5.4 - Support pagination parameters within AWS client.
• 5.5.3 - Adding in a new endpoint that can be called to run a task connection test
• 5.5.2 - Address bug with typing for type List in Python 3.8
• 5.5.1 - Address bug with typing for type Tuple in Python 3.8
• 5.5.0 - Updated helper class to add make_request, response_handler, extract_json, and request_error_handling for HTTP requests, and hash_sha1 and compare_and_dedupe_hashes to provide support for hash comparisons | Add METHOD_NOT_ALLOWED, CONFLICT, REDIRECT_ERROR, and CONNECTION_ERROR to PluginException presets
• 5.4.9 - Updated aws_client to clean assume role json object to remove any none or empty string values.
• 5.4.8 - Address vulnerabilities within gunicorn and idna python packages.
• 5.4.7 - Address vulnerabilities within insightconnect-python-3-plugin image.
• 5.4.6 - Updated our cloud loggers to include the request path | Updated slim image to remediate vulnerabilities.
• 5.4.5 - Updated base images to pull Python 3.9.19 | Alpine image: Updated expat package
• 5.4.4 - Handling the 'Connection Refused' to cps (during startup) as an info instead of error log as this is an expected error.
• 5.4.3 - Updated dependencies to the latest versions | Fixed issue related to logger StreamHandlers
• 5.4.2 - Remove background scheduler to simplify when the server gets custom_config values | Revert to use bullseye for slim image.
• 5.4.1 - Retry logic added to get values from external API for custom_config values.
• 5.4.0 - Implementation of custom_config parameter for plugin tasks | Alpine image updated OpenSSL and expat | Use bookworm for slim image.
• 5.3.2 - Updated OpenSSL in alpine image and core packages to latest versions.
• 5.3.1 - New logging added to the beginning and end of a task | New logging when an exception is instantiated.
• 5.3.0 - Update base images to pull Python 3.9.18 | python packages bump | rename image to drop python minor version.
• 5.2.4 - Extended logging with OrgID and IntID
• 5.2.3 - Extended logging in AWSClient
• 5.2.2 - Fix longstanding bug where some error responses from a plugin could be in HTML format instead of JSON
• 5.2.1 - Add INVALID_CREDENTIALS exception preset
• 5.2.0 - Add status_code and exception properties to task output
• 5.1.4 - Fix credential masking when connection is null
• 5.1.3 - Fix credential masking when some input fields are empty
• 5.1.2 - Add connection credential masking to the plugin's output and log when displayed as plain text | Add new OutputMasker class to handle credential masking
• 5.1.1 - Updated exception preset messages
• 5.1.0 - Add new helper functions
• 5.0.0 - Add has_more_pages property to task output to indicate task pagination status to output consumers
• 4.10.1 - Remove raising of exception if request id is not available in header
• 4.10.0 - Add structlog for structured logging
• 4.9.0 - Add current SDK version plugin is using to /info endpoint
• 4.8.0 - Add OAuth20ClientCredentialMixin class to clients
• 4.7.6 - Add PaginationHelper to the AWS Client | Refactored the ActionHelper | Add region handler for AWSAction
• 4.7.5 - Add AWS client for assuming role
• 4.7.4 - Convert data field to string in exception handling
• 4.7.3 - Add measurement_time property to plugin metrics collection
• 4.7.2 - Fix incorrect status codes when handling PluginExceptions
• 4.7.1 - Fix new connection test endpoint, version pin requests to 2.26.0
• 4.7.0 - Add endpoint for retrieving all action definitions (input schemas for all actions within a plugin)
• 4.6.0 - Add Bad Response preset to PluginException | Improve 400 error exception handling
• 4.5.1 - Bump version number for release pipeline
• 4.5.0 - Improve exception handling for non-ConnectionTestException errors during a connection test
• 4.4.0 - Add initial support for plugin runtime metrics collection
• 4.3.3 - Constrain greenlet dependency version to fix conflict with gevent
• 4.3.2 - Update Flask dependency to version 2.0.3
• 4.3.1 - Add Timeout Preset to ConnectionTestException
• 4.2.1 - Add helpful error messages in JSON
• 4.2.0 - Add implementation and endpoints for tasks
• 4.1.1 - Update gevent dependency version to 20.9.0
• 4.1.0 - Provide ability to run with gevent asynchronous gunicorn worker class for increased performance
• 4.0.3 - Fix to avoid command injections when using exec_command helper method
• 4.0.2 - Fix to remove unprintable characters from trigger logs
• 4.0.1 - Fix bug by including schema files in manifest | Fix issue uploading python library twice to PYPI
• 4.0.0 - Implement new API endpoints | Implement Swagger API documentation generation | End support for Python2 and PyPy | Add development details to README | Enhancements to Makefile for local development and release | Rebrand SDK to InsightConnect Python Plugin Runtime | Revamp release process with use of Github Actions
• 3.3.0 - Add webserver route to allow for threading changes
• 3.2.0 - Add new ConnectionTestException/PluginException presets: UNKNOWN, BASE64_ENCODE, BASE64_DECODE, INVALID_JSON | Add an optional data parameter for formatting response output