ws-cleanup-tool-test 23.6.1.3

Mend SCA Cleanup Tool

Mend Projects Cleanup CLI Tool

• The self-hosted CLI tool features cleaning up projects and generating reports before deletion in 2 modes:
  • By stating OperationMode=FilterProjectsByUpdateTime and how many days to keep (-r/ DaysToKeep=)
  • By stating OperationMode=FilterProjectsByLastCreatedCopies and how many copies to keep (-r/ DaysToKeep=)
• The reports are saved in the designated location as follows: [Output_DIR]/[PRODUCT NAME]/[PROJECT NAME]/[REPORT NAME]
  • The default location is the [WORKING DIRECTORY]/Mend/Reports/[PRODUCT NAME]/[PROJECT NAME]/[REPORT NAME]
• To review the outcome before actual deletion use -y true / DryRun=True flag. It will NOT delete any project nor create reports
• By default, the tool generates all possible project-level reports. By specifying ((-t / Reports=/) it is possible to select specific reports
  • The full list of available reports is below
• The full parameters list is available below
• There are two ways to configure the tool:
  • By configuring params.config on the executed dir or passing a path to the file in the same format
  • By setting command line parameters as specified in the usage below

Supported Operating Systems

• Linux (Bash): CentOS, Debian, Ubuntu, RedHat
• Windows (PowerShell): 10, 2012, 2016

Pre-requisites

• Python 3.8+

Permissions

• The user used to execute the tool has to have "Organization Administrator" or "Product Administrator" on all the maintained products and "Organization Auditor" permissions.
• It is recommended to use a service user.

Installation and Execution from PyPi (recommended):

1. Install by executing: pip install mend-sca-cleanup-tool
2. Configure the appropriate parameters either by using the command line or in params.config.
3. Execute the tool (mend_sca_cleanup_tool ...).

Installation and Execution from GitHub:

1. Download and unzip mend-sca-cleanup-tool.zip from the most recent tagged release.
2. Install requirements: pip install -r requirements.txt
3. Configure the appropriate parameters either by using the command line or params.config.
4. Execute: python sca_cleanup_tool.py <CONFIG_FILE>

Examples:

Perform dry run check-in to get to know which projects would have been deleted:
mend_sca_cleanup_tool -r 30 -m FilterProjectsByUpdateTime -u <USER_KEY> -k <ORG_TOKEN> -y true

---

Keep the last 60 days on each product, omitting a product token <PRODUCT_1> from analyzing:
mend_sca_cleanup_tool -r 60 -m FilterProjectsByUpdateTime -u <USER_KEY> -k <ORG_TOKEN> -e <PRODUCT_TOKEN_1>

---

Keep only two of the newest projects in each product token PRODUCT_1 and PRODUCT_2:
mend_sca_cleanup_tool -r 2 -m FilterProjectsByLastCreatedCopies -u <USER_KEY> -k <ORG_TOKEN> -i <PRODUCT_TOKEN_1>,<PRODUCT_TOKEN_2>

---

Analyze only the projects that have the specified Mend tag and keep the newest project in each product:
mend_sca_cleanup_tool -r 1 -m FilterProjectsByLastCreatedCopies -u <USER_KEY> -k <ORG_TOKEN> -g <KEY>:<VALUE>

---

Keep the last 2 weeks and analyze only the projects whose match their tag key and the tag value contains the specified value:
mend_sca_cleanup_tool -r 14 -m FilterProjectsByUpdateTime -u <USER_KEY> -k <ORG_TOKEN> -v <KEY>:<VALUE>

---

Keep the last 100 days for both PRODUCT_1 and PRODUCT_2, but do not delete the project PROJECT_1 (which is a project in one of the included products):
mend_sca_cleanup_tool -r 100 -m FilterProjectsByUpdateTime -u <USER_KEY> -k <ORG_TOKEN> -i <PRODUCT_TOKEN_1>,<PRODUCT_TOKEN_2> -x <PROJECT_TOKEN_1>

---

Keep the last month for both PRODUCT_1 and PRODUCT_2, but do not delete projects that contain provided strings in their names:
mend_sca_cleanup_tool -r 31 -m FilterProjectsByUpdateTime -u <USER_KEY> -k <ORG_TOKEN> -i <PRODUCT_TOKEN_1>,<PRODUCT_TOKEN_2> -n CI_,-test

---

Full Usage flags:

usage: ws_cleanup_tool [-h] -u MEND_USER_KEY -k MEND_TOKEN [-a MEND_URL] [-t REPORT_TYPES] [-m {FilterProjectsByUpdateTime,FilterProjectsByLastCreatedCopies}] [-o OUTPUT_DIR] [-e EXCLUDED_PRODUCT_TOKENS] [-i INCLUDED_PRODUCT_TOKENS]
                    [-g ANALYZED_PROJECT_TAG] [-r DAYS_TO_KEEP] [-p PROJECT_PARALLELISM_LEVEL] [-y DRY_RUN]

Mend Cleanup Tool

optional arguments:
  -h, --help            show this help message and exit
  -u MEND_USER_KEY, --userKey 
                    Mend User Key
  -k MEND_API_TOKEN, --apiToken, --orgToken
                    Mend Organization Key (API Key)
  -a MEND_URL, --mendUrl, --wsURL
                    Mend URL. This value defaults to saas.whitesourcesoftware.com.
  -t REPORT_TYPES, --reportTypes
                    Report Types to generate (comma seperated list)
  -m OPERATION_MODE, --operationMode {FilterProjectsByUpdateTime,FilterProjectsByLastCreatedCopies}
                    Cleanup operation mode
  -o OUTPUT_DIR, --outputDir
                    Output directory
  -e EXCLUDED_PRODUCT_TOKENS, --excludedProductTokens
                    List of excluded products
  -i INCLUDED_PRODUCT_TOKENS, --includedProductTokens
                    List of included products
  -g ANALYZED_PROJECT_TAG, --AnalyzedProjectTag
                    Analyze only the projects whose contain the specific Mend tag (key:value). Case sensitive.
  -v ANALYZED_PROJECT_TAG_REGEX_IN_VALUE, --AnalyzedProjectTagRegexInValue
                    Analyze only the projects whose match their tag key and the tag value contains the specified value (key:value). Case sensitive.
                    Note: This was originally broken in the original ws-cleanup-tool. The functionality was adjusted to work as originally written. The naming convention is a misnomer but was kept to avoid breaking existing integrations.
  -r DAYS_TO_KEEP, --DaysToKeep
                    Number of days to keep in FilterProjectsByUpdateTime or number of copies in FilterProjectsByLastCreatedCopies
  -p PROJECT_PARALLELISM_LEVEL, --ProjectParallelismLevel
                    Project parallelism level
                    Note: This is currently not used in this version of the mend-sca-cleanup-tool. Was kept to prevent breaking existing integrations.
  -y DRY_RUN, --DryRun
                    Logging the projects that are supposed to be deleted without deleting and creating reports
                    default False
  -s SKIP_REPORT_GENERATION, --SkipReportGeneration
                    Skip report generation step
                    default False
  -j SKIP_PROJECT_DELETION, --SkipProjectDeletion
                    Skip project deletion step
                    default False                                        
  -x EXCLUDED_PROJECT_TOKENS, --excludedProjectTokens
                    List of excluded projects
  -n EXCLUDED_PROJECT_NAME_PATTERNS, --excludedProjectNamePatterns
                    List of excluded project name patterns (comma seperated list). Case sensitive.            

Available reports

The following Mend project reports are available through the clean-up tool. These values can be specified with the -t flag to generate specific reports.

• alerts
• alerts_rejected_by_policy
• attribution
• bugs
• due_diligence
• ignored_alerts
• in_house_libraries
• inventory
• license_compatibility
• resolved_alerts
• request_history
• source_files
• source_file_inventory
• vulnerability

SAST Clean up

If you need to run a clean up script for your SAST environment, please refer to the Mend SAST clean up kit in the Mend Toolkit

note: The optimal cleanup scope is derived from the size of the environment, Mend scope size (memory and CPU) allocated for the server, and runtime time constraints.