scantist-karby 4.2.0

scan the provided projects by using whitesource or scantist, and convert the result into scantist's format

Karby

Prerequisite

1. install pipenv according to any tutorial.
2. enter Karby main folder, execute pipenv shell, run pipenv install to install the dependencies.
3. There is a filed named .env.example, copy one and change it to .env . This is where we put all the system environment to. Every time you initiate the pipenv shell, the system will read those environment variables. All the credentials, tokens or password should not be directly placed in code, but here in .env

Running

now you should be able to run the script.

The entrance is sca_tool_scan.py. the full command example is

python sca_tool_scan.py [ossindex|scantist|whitesource] [upload|cmd] <any/git/url> -name <this is optional> -output <this is optional>

you may not be able to run this because you don't have snyk credentials in your environment

for parameter details, you should run python sca_tool_scan.py -h

Basic Idea

Motivation

Basically, vulnerability detection tools are aim to find all the components in a given project as well as their vulnerabilities. But different tools are so different in using. karby is a project aims to collect all of these vulnerability detection tools and simplify their usage.

Input and output

The input should be a single project URL. It could be a git hub URL(for upload scan) or local project directory(for cmd scan).

The output should be a CSV file in the format of Scantist. You could see the example in karby/report_format_example . There should be 2 files, the format is <tool_name>-<component or issue>-<project name>.csv.

• component file: there are 10 fields in this report, but actually only 2 of them matters: Library and Version. Just leave the rest empty
• issue file: there are 10 fields in this report, but actually only 3 of them matters: Library Library Version and Public ID. Public ID is the CVE public Id, if it is not provided, you can also left it empty.

Scan types

I defined 2 types of scan: cmd and upload. They are also called: ci scan and airgap scan.

• cmd scan need the user to provide the local well-build project directory, and trigger the scan locally in command line. Then, you can use any method to collect the result. You can download the report by url or just parse the output from command line. We don't care, but keep it simple and guarantee the final format(Scantist format)
• upload scan need the user to provide the github url to trigger the scan. Or say, trigger by calling APIs given by that tool. This method do not require local build, so we call this **un-build ** project scan as airgap scan. Not all tools support this upload scan method. Need an example? You can refer to snyk api scan.

Scan Tools

Whitesource

Mode avaliable

• cmd: support
• upload: support (this will not trigger a new scan, but find the project online by given name)

Prerequisite

1. One of the following Java versions:
   • Java JDK 8
   • Java JRE 8
   • Java JDK 11
2. Depending on your project type, ensure that the relevant package managers are installed. You can refer to "Prerequisites" section in the following link for more details.
   • https://whitesource.atlassian.net/wiki/spaces/WD/pages/1140852201/Getting+Started+with+the+Unified+Agent

Setup

1. Download wss-unified-agent.jar from the following links:
   • https://unified-agent.s3.amazonaws.com/wss-unified-agent.jar

Environment Variables

• WHITESOURCE_API_KEY: WhiteSource API key, a unique identifier of your organization. Should be a 256-bit hex number.
• WHITESOURCE_USER_KEY: WhiteSource User Key. Can be generated from the Profile page in WhiteSource account. In order to get a scan report, the user key must be generated by administrators. Should be a 256-bit hex number.
• WHITESOURCE_PRODUCT_NAME: Name of your product.
• PATH_TO_WHITESOURCE_JAR (cmd mode only): Path to the Whitesource local agent .jar file (e.g. /path/to/jar/wss-unified-agent.jar)
• PATH_TO_WHITESOURCE_CFG (cmd mode only, optional): Path to the Whitesource local agent .config file (e.g. /path/to/cfg/wss-unified-agent.cfg). We put a prepared one inside folde resources. If you need any special modification, go to resources/wss-unified-agent.config

snyk (deprecated)

Mode avaliable

• cmd: support
• upload: support

Prerequisite

1. install snyk command line tool in the local environment using

   npm install -g snyk
   

   https://support.snyk.io/hc/en-us/articles/360003812538-Install-the-Snyk-CLI#UUID-b346a6dc-682a-471f-bdc2-a82d8e5f8b6e

2. see if snyk is successfully installed

   snyk version
   

Environment Variables

• SNYK_USR_TOKEN: can be retrieved from personal setting: https://app.snyk.io/manage/integrations
• SNYK_ORG_ID: can be retrieved from personal setting
• SNYK_INTEGRATION_ID: can be retrieved from personal setting

scantist

Mode avaliable

• cmd: support
• upload: not support

Prerequisite

1. download scantist-bom-detect.jar from https://scripts.scantist.com/staging/scantist-bom-detect.jar

Environment Variables

• SCANTIST_EMAIL: scantist account email
• SCANTIST_PSW: scantist account password
• SCANTIST_BASEURL: target base url, default to "https://api-staging.scantist.io/"
• SCANTIST_SBD_HOME: path to scantist-bom-detect.jar