Text Similarity

Tool to identify the similarity of the input text

It can be used to identify the similarity of,

• Tests

• Code

• Requirements

• Defects

Advantage of using such similarity analysis are,

• Resolving technical debt

• Grouping together similar code / tests / requirements / defects etc.

Dependencies

• python 3.8 : 64 bit

• python packages (xlrd, xlsxwriter, pandas, scikit-learn, numpy)

Installation

Requirements are added in requirement.txt file

Python 3.8

python: install python for the respective OS at https://www.python.org/downloads/ Make sure to update the path variable to point to the python installation folder.

pip: (only if pip is not present by default) get get-pip.py from below link to your folder https://bootstrap.pypa.io/get-pip.py Open a command prompt and navigate to the folder containing get-pip.py. Run the following command:

similarity-processor:

pip install similarity-processor

pip install similarity-processor

Usage

UI

>>>python -m similarity.similarity_ui

• Path to the test/requirement/other other document to be analyzed(xlsx / csv format).

• Unique ID in the csv/xlsx column ID(0/1 etc...)

• Steps/Description id for content matching (column of interest IDs in the csv/xlsx separated by , like 1,2,3)

• If new requirement / test to me checked with existing, enable the check box and paste the content to be checked in the new text box.

Commandline

>>>python -m similarity --p "path\to\TestBank.xlsx" --u 0 --c "1,2,3" --n 8

• Help option can be found at,

>>>python -m similarity --h

Code

>>> from similarity.similarity_io import SimilarityIO
>>> similarity_io_obj = SimilarityIO("path\to\TestBank.xlsx", 0, "1,2,3")
>>> similarity_io_obj.orchestrate_similarity()

Arguments

Mandatory

• Path to the input file
• Unique id value column id in xlsx
• Interested columns in xlsx

Optional

• Upper and lower range to filter the similarity values in the output (defaulted "60,100")
• Number of rows in the html report, defaulted to 100
• Are you checking a new text against a existing text bank?
• If yes: new text
• Filter value to split the report xlsx file, defaulted to 500000, 500001 onward row will be moved to new file

import pandas as pd
from similarity.similarity_io import SimilarityIO

demo_df = pd.read_excel(r"input\xlsx\sheet\name")  # You could read from any input source

similarity_io_obj = SimilarityIO(None, None, None)  # (None, None, None, 200) =>200 = The brief html report rows
 default is 10  
similarity_io_obj.file_path = r"path\to\report\folder" #when used in this format, else input file path to read data
similarity_io_obj.data_frame = demo_df # input data frame
similarity_io_obj.uniq_header = "Uniq ID"  # Unique header of the input data frame (string)
similarity_io_obj.create_merged_df()
processed_similarity = similarity_io_obj.process_cos_match()
similarity_io_obj.report_brief_html(processed_similarity)
processed_similarity.to_csv(r"path\to\report\folder\report.csv", header=True)

Output

• Output will be available in same folder as input file or file_path specified

• If any duplicate ids in the unique id file with name string containing 'duplicate id'

• A recommendation file with similarity values

• A merged file with data in the "interested columns in xlsx"

• An html brief report containing the top 10 similarities (100 is default value which can be changed by --n option)

Contact

Brijesh Krishnan brijesh.krishnank@philips.com Dattatreya Vellal dsvellal@philips.com

License

The MIT License (MIT) Copyright © [2019] Koninklijke Philips N.V, https://www.philips.com

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.