blackduck-lutils 0.0.19

Collection of license related tools. License-tagging, License-conflicts

Black Duck License Utilities

This module delivers functionality for

• License Conflicts
• License Tagging and Grouping

Requirements

Module requires Python 3.8 or higher. The following modules are required:

• argparse
• blackduck
• openpyxl

Installation

Module is installed using pip command

pip3 install blackduck-lutils

License Conflicts

Description

This functionality will generate license conflict report by comparing license terms of the licenses associated with components on the Bill of Materials. Project version license will be taken into account and license conflicts for it will be calculated as well.

How it works

License conflict computation process based on license terms associated with each license. License terms can be

• Permitted
• Required
• Prohibited

This script will calculate instances when a licenses associated with components included in the project require and prohibit the same terms respectively. E.g. One license requires Commercial Use and another one prohibits Commercial Use thus creating conflict

The list of standard license terms defined by Black Duck is as following:

License Term	Responsibility	Description
Anti DRM Provision	FORBIDDEN	You are not allowed to use this software in conjunction with Digital Rights Management technology or technological protection measures
Commercial Use	PERMITTED	License permits software to be used commercially
Commercial Use	REQUIRED	License requires software to be used for commercial purposes only
Commercial Use	FORBIDDEN	License forbids or places restrictions on commercial use of the software
Compensate Damages	PERMITTED	If software is part of a commercial product, you may defend and compensate project contributor from lawsuits & damages caused by your commercial offering, but are not required to do so
Compensate Damages	REQUIRED	If software is part of a commercial product, you must defend and compensate project contributor from lawsuits & damages caused by your commercial offering
Compensate Damages	FORBIDDEN	If software is part of a commercial product, license states you cannot defend and compensate project contributor from lawsuits & damages caused by your commercial offering
Disclose Source	REQUIRED	Software source code must be made available if a distribution is made
Disclose Source	FORBIDDEN	License forbids you from making the software source available if a distribution is made
Disclose Source	PERMITTED	Software source code can be made available if a distribution is made, but you are not required to do so
Distribute	REQUIRED	If distributed, you are required to distribute unmodified software
Distribute	PERMITTED	This software may be distributed in original form or modified works
Distribute	FORBIDDEN	This software may not be distributed in original form or modified works
Distribute Original	REQUIRED	Upon distribution, you will need to provide either the original software source code, a link to the original software or provide a written offer to receive the software
Distribute Original	PERMITTED	Upon distribution, you are allowed to provide either the original software source code, a link to the original software or provide a written offer to receive the software
Distribute Original	FORBIDDEN	Upon distribution, you are forbidden to provide either the original software source code, a link to the original software or provide a written offer to receive the software. You may distribute binaries only
Fees	FORBIDDEN	You are not allowed to charge the recipient certain fees
Fees	REQUIRED	Recipient is expected to pay a fee
Give Credit	FORBIDDEN	License forbids you to acknowledge the original author of the software in distributions you make
Give Credit	PERMITTED	License allows you to acknowledge the original author of the software in distributions you make, but you are not required to do so
Give Credit	REQUIRED	License requires you to acknowledge the original author of the software in distributions you make
Hold Liable	REQUIRED	License requires you to issue a disclaimer of warranty on your code which prevents you or anyone using your code from holding the original owner and/or contributors liable for any damages due to the use of this software
Hold Liable	FORBIDDEN	A disclaimer of warranty prevents you from holding the original owner and/or contributors liable for any damages due to the use of this software
Hold Liable	PERMITTED	License does not contain a disclaimer of warranty which prevents you from holding the original owner and/or contributors liable for any damages due to the use of this software
Include Copyright	PERMITTED	You may include copyright notices of the original owners and contributors for the work with the software you distribute
Include Copyright	REQUIRED	You are required to include copyright notices of the original owners and contributors for the work with the software you distribute
Include Copyright	FORBIDDEN	You are forbiddeen to include copyright notices of the original owners and contributors for the work with the software you distribute
Include Install Instructions	PERMITTED	If the software is distributed, you may include installation information and tools to modify and reinstall the software, but you are not required to do so
Include Install Instructions	REQUIRED	If the software is distributed, you must include installation information and tools to modify and reinstall the software
Include Install Instructions	FORBIDDEN	If the software is distributed, you are forbidden to include installation information and tools to modify and reinstall the software
Include License	PERMITTED	You are permitted to include the software's full license text in any distributions you make, but not required to do so
Include License	FORBIDDEN	You are forbidden to include the software's full license text in any distributions you make
Include License	REQUIRED	You are required to include the software's full license text in any distributions you make
Include Notice	PERMITTED	You are permitted to include a notice in product documentation with certain requirements as specified by the license, but not obligated to do so
Include Notice	FORBIDDEN	You are forbidden to include a notice in product regarding the use of this component
Include Notice	REQUIRED	You are required to include a notice in product documentation with certain requirements as specified by the license
License Back	REQUIRED	You are required to license any modifications back to the original licensor
Modify	FORBIDDEN	You are forbidden to modify the software and make derivative bodies of work
Modify	PERMITTED	You are allowed to modify the software and make derivative bodies of work
Modify	REQUIRED	You are required to grant the right to modify the software and make derivative bodies of work
Patent Retaliation	FORBIDDEN	You are not allowed to file a patent suit against a specific person/organization or related to the work
Place Additional Restrictions	FORBIDDEN	You are not entitled to place additional restrictions
Place Additional Restrictions	PERMITTED	You intend to place additional restrictions
Place Warranty	PERMITTED	You are allowed to place a warranty on the code or derivatives of the code, though you may need to limit the liability of the original software owners and contributors
Place Warranty	FORBIDDEN	This license contains restrictions that forbid or severely limits your ability to place a warranty on the code or derivatives of the code
Place Warranty	REQUIRED	You are required to place a warranty on the code or derivatives of the code, though you may need to limit the liability of the original software owners and contributors
Private Use	REQUIRED	This software requires that it only be used for private use/non-commercial purposes
Private Use	PERMITTED	You are allowed to use this software for private, non-commercial use without payment or royalty
Private Use	FORBIDDEN	You are forbidden to use this software for private, non-commercial use without payment or royalty
Rename	REQUIRED	If modified, you are required to rename the software to indicate it is not the original work
Rename	PERMITTED	If modified, you are allowed to rename the software
Rename	FORBIDDEN	If modified, you are forbidden to rename the software
Reverse Engineer	FORBIDDEN	You are not allowed to reverse engineer the software
Reverse Engineer	REQUIRED	You are required to grant the right to reverse engineer the software
Right to Copy	FORBIDDEN	You are not entitled to grant the right to copy the code
Right to Copy	REQUIRED	You are required to grant the right to copy the code
State Changes	FORBIDDEN	Any changes you make to the code must not be documented and distributed along with the code
State Changes	REQUIRED	Any changes you make to the code must be documented and distributed along with the code
State Changes	PERMITTED	Any changes you make to the code may be documented and distributed along with the code, but you are not required to do so
Sub-License	REQUIRED	You are required to provide the software or modifications to the software under a different license
Sub-License	FORBIDDEN	You may not provide this software or modifications under a different license
Sub-License	PERMITTED	You may provide the software or modifications to the software under a different license
Use Patent Claims	FORBIDDEN	You are expressly forbidden to use the relevant patents held by the software contributors in your use of this code without a separate patent license
Use Patent Claims	REQUIRED	License explicitly states you (or others who use this software) are required to purchase a Patent license to use this software
Use Patent Claims	PERMITTED	You may use the relevant patents held by the software contributors in your use of this code
Use Trademarks	PERMITTED	License explicitly states that it grants you rights to use the owners or contributors trademarks
Use Trademarks	REQUIRED	License explicitly states that it requires you to use to use the owners or contributors trademarks
Use Trademarks	FORBIDDEN	License explicitly states that it does not grant you any rights to use the owners or contributors trademarks

Custom license terms added to the system will be included in conflicts calculations.

License conflict calculations will take into account relationships between components and automatically exclude the following categories from conflict calculations:

• Development Tools
• Merely Aggregated
• Implementation of Standard
• Unspecified

Unknown licenses will be ignored and not included into conflict calculations.

The process is designed to minimize API calls required to produce a report. This is achieved by putting license information into a local storage file and utilizing SBOM report as the data source for project information.

License conflicts defines two main functions

• download_license_data - used to download license data during bootstrapping process
• generate_license_conflict_report - core functionality

Module provides main() function to allow either direct execution or integrating into larger automation.

Bootstrapping

Before using the script, license data would have to be downloaded from the system and stored in to a file. To perform that task using direct execution perform the following:

python3 -m blackduck_lutils.license_conflicts -u $BD_URL -t token_file -nv -ldd [-ld license_data.json]

This will download license data into 'license_data.json` file in the current directory. To place the file into different location use corresponding command line option.

Same task can be accomplished with the following code:

from blackduck_lutils import license_conflicts

license_conflicts.download_license_data(base_url=blackduck_url, 
                                        token=token, 
                                        no_verify=no_verify_flag, 
                                        output_file = license_data_filename)

Note: This operation will execute about 3000 API calls and that is primarily the reason why we put this data into local storage. This way we don't have to put that much load onto the server when generating conflict reports.

Note: This file would have to be regenerated if new licenses are added to the system.

Generating Conflict Report

This utility cam be used in off-line and on-lne modes.

In-line function

Tis function is structured to be included in automation scripts. It will accept Client object as a parameter and will return python object as a result set. It will not write result into a file. The following actions will take place:

• Issue an API request to generate SBOM report.
• Wait for report completion
• Download the report
• Generate license conflicts data and return it

Inline function supports optional parameter to exclude project license from conflict calculations.

Generating License Conflict Report with inline function can be accomplished as following:

from blackduck_lutils import license_conflicts

license_conflicts = license_conflicts.generate_license_conflict_report(bd=bd,
                                                project_name=args.project_name,
                                                project_version_name=args.project_version_name,
                                                license_data_file=args.license_data,
                                                fail_unknown_licenses=True|False,
                                                exclude_project_license=False|True)
                                                  

license conflict data will contain a list of license conflict pairs structured as following:

[
  . . .
  {
    'Category': 'Category',
    'ComponentOne': 'Name of the first component in conflicting pair',
    'ComponentOneLicense']: 'License of the first component',
    row['ComponentTwo']: 'Name of the second component in conflicting pair',
    row['ComponentTwoLicense']: 'License of the second component',
    row['LicenseTerm']: 'Conflicting term name',
    row['TermValueOne']: 'Conflicting term value for the first component',
    row['TermValueTwo']: 'Conflicting term value for the second component',
    row['TermDescriptionOne']: 'Long description for the conflicting term value for the first component',
    row['TermDescriptionTwo']: 'Long description for the conflicting term value for the second component'
  }
  . . .
]

On-line mode

In on-line mode the following actions will take place:

• Issue an API request to generate SBOM report.
• Wait for report completion
• Download the report
• Generate license conflicts data and write it into CSV file

To generate License Conflict Report in on-line mode using direct execution perform the following:

python3 -m blackduck_lutils.license_conflicts -u $BD_URL -t token -nv -pn ProjectName -pv ProjectVersion \
                                              [-ld license_data.json] [-o output_file]

Where $BD_URL is the URL of your Black Duck system and 'token' is a file containing API token.

Same task can be accomplished with the following code:

from blackduck_lutils import license_conflicts

license_conflicts.generate_license_conflict_report(base_url=blackduck_url, 
                                                  token=token_file, 
                                                  no_verify=no_verify_flag, 
                                                  project_name=project_name, 
                                                  project_version_name=project_version_name,
                                                  license_data_file=license_data_file,
                                                  csv_report_file=output_file,
                                                  fail_unknown_licenses=True|False,
                                                  xls=True|False)

Off-line mode

In off-line mode a SBOM report in SPDX format would have to exist. The following actions will take place:

• Load SBOM report from a file
• Generate license conflict data and write it into a file

To generate License Conflict Report in off-line mode using direct execution perform the following:

python3 -m blackduck_lutils.license_conflicts -u $BD_URL -t token -nv -sbom SBOM_FILE \
                                              [-ld license_data.json] [-o output_file]

Where $BD_URL is the URL of your Black Duck system and 'token' is a file containing API token.

Same task can be accomplished with the following code:

from blackduck_lutils import license_conflicts

license_conflicts.generate_license_conflict_report(sbom=SBOM_FILE, 
                                 license_data_file=license_data_file,
                                 csv_report_file=output_file,
                                 fail_unknown_licenses=True|False,
                                 xls=True|False))

Module Command Line specification

$ python3 -m blackduck_lutils.license_conflicts -h
usage: license_conflicts [-h] -u BASE_URL -t TOKEN_FILE [-nv] [-ldd] [-pn PROJECT_NAME]
                                 [-pv PROJECT_VERSION_NAME] [-sbom SBOM] [-ld LICENSE_DATA] [-o OUTPUT_FILE]

options:
  -h, --help            show this help message and exit
  -u BASE_URL, --base-url BASE_URL
                        Hub server URL e.g. https://your.blackduck.url
  -t TOKEN_FILE, --token-file TOKEN_FILE
                        File containing access token
  -nv, --no-verify      Disable TLS certificate verification
  -ldd, --license-data-download
                        Download license data into a file specified with -ld/--license-data parameter
  -pn PROJECT_NAME, --project-name PROJECT_NAME
                        Project Name
  -pv PROJECT_VERSION_NAME, --project-version-name PROJECT_VERSION_NAME
                        Project Version Name
  -sbom SBOM            SBOM File to process
  -ld LICENSE_DATA, --license-data LICENSE_DATA
                        Local license data storage
  --fail-unknown-licenses
                        Stop processing if Unknown licenses are present
  --skip-terms SKIP_TERMS
                        License terms to exclude from analysis. Comma separated list
  --print-defined-terms
                        Print all defined license terms and exit
  --xls                 Use EXCEL format instead of CSV
$

Results

Conflict report will be written into CSV file with one line per license term conflict. The following fields will be present in the file:

• Category
• ComponentName
• ComponentLicense
• ConflictingComponentName
• ConflictingComponentLicense
• ConflictingLicenseTerm
• TermValue
• ConflictingTermValue
• TermDescription
• ConflictingTermDescription

TODO

This utility is tested in off-line and online mode and has sufficient functionality for API performance testing.

It is not fully tested against functional specification yet.

License

This utility is licenses under Apache 2.0 License.

License Tagging and Grouping

License tagging and grouping to allow easy insertion of arbitrary groups of licenses into policy definitions.

Description

When there is a need to add an arbitrary group of licenses into a Black Duck policy it requires adding them one by one through the UI. This tool allows to maintain an external data store with Tags ang Tag mappings to the licenses and will allow updating policy definitions with expanded list of licenses.

Usage

Tool can be executed as a direct module execution with the following command line specification:

$ python3 -m blackduck_lutils.license_tagging -h
usage: license_tagging [-h] -u BASE_URL -t TOKEN_FILE [-nv] [-lsf LICENSE_STORE_FILE] [-nlc]

options:
  -h, --help            show this help message and exit
  -u BASE_URL, --base-url BASE_URL
                        Hub server URL e.g. https://your.blackduck.url
  -t TOKEN_FILE, --token-file TOKEN_FILE
                        File containing access token
  -nv, --no-verify      Disable TLS certificate verification
  -lsf LICENSE_STORE_FILE, --license-store-file LICENSE_STORE_FILE
                        Local license information storage
  -nlc, --no-license-check
                        Skip scanning trough all licenses

### How it works
The tool maintains local local data storage as an Excel file with default filename of 'LicenseStorage.xlsx'

This file will contain two worksheets
- Licenses
- LicenseTags

Licenses worksheet contains data on all licenses currently present in Black Duck instance. Columns contain data and are used as following:
* Column A - License Name
* Column B - License URL
* Column C - License Family
* Column D - License Family URL
* Column E - SPDXID
* Column F and above can contain tags

LicenseTags worksheet contains License Tag Names in the Column A and corresponding URL in column B. 

#### Common parameters
In the example command line there would be references to common parameters:
* $BD_URL - environment variable containing Black Duck instance URL
* token. - a text file containing valid authentication token

#### Bootstrapping
Before using the tool, local storage would have to be initialized.
That is accomplished by running the script.  If it can not find local license store, it will generate new file with complete internal structures and populate Licenses worksheet with data from Black Duck instance.

python3 -m blackduck_lutils.license_tagging -u $BD_URL -t token -nv

This will generate a file named LicenseStorage.xlsx in the current folder and populate Licenses worksheet with full complement of licenses from your Black Duck instance.

It will also validate presence of 'TagPlaceholder' and create it as necessary.
At this point there will be no tags defined yet.

#### Defining tags
User must open the local store excel file and add tags to Column A of LicenseTag worksheet, save the Excel file and run the script again.

python3 -m blackduck_lutils.license_tagging -u $BD_URL -t token -nv

This action will update the references in Black Duck and get the framework ready to be used.

#### Tagging Licenses
Using Excel as a tool, tag licenses on the License worksheet by putting tag name in the columns F and above. There is no limit on how many tags could be associated with a license. 

#### Policy creation
When creating a policy that would have to reference large number of licenses use TagName which will be visible and searchable in the UI.

#### Expanding tags to full list of licenses
Execute the script again and the Tags in the policy will be expanded to full list of licenses.

python3 -m blackduck_lutils.license_tagging -u $BD_URL -t token -nv

This action will iterate through the policies, find policy conditions that have tag name in them and replace them with corresponding list of licenses.

### Example of use
See example of use here

### License
This code is licensed under Apache 2.0 license.

### Project status
Active