spacy-matching

Background

In Germany's cancer registries, substances are reported in a free text field, e.g., "Interferon alpha-2a weekly i.v.". The reported text might include additional information such as dosage or labels or typos. For analysis, it is helpful to have substances reported in a harmonized way. The aim of this Python function is extracting the substance name from the free text field. The script uses the FuzzyMatcher from spaczz and spacy to scan each free text for potential matches.

Usage

Call add_substance() with these arguments as Series:

• col_with_substances - original substance name (freetext)
• col_with_ref_substances - reference list of substances

Returns a dataframe with the following columns:

• Original - column with original value
• Mapped_to - column with recoded value, i.e the substance name that is deemed most likely
• Similarity - column with similarity score

Original                   Mapped_to         Similarity
0            Fluorouracil  Fluorouracil      1.00
1              Epirubicin  Epirubicin        1.00
2        PEG-Asparaginase  Pegaspargase      0.86
3             Trastuzumab  Trastuzumab       1.00
4  Temozolomid (Temodal®)  Temozolomid       1.00

from spacy_matching import recoding as rec

# * get substance reference table from public repo
URL_V2 = "https://gitlab.opencode.de/robert-koch-institut/zentrum-fuer-krebsregisterdaten/cancerdata-references/-/raw/main/data/v2/Klassifikationen/substanz.csv?ref_type=heads"
reference_list = pd.read_csv(URL_V2, sep=";")

# * create a pandaSeries with some test data
fake_data = pd.Series(["Interferon alpha 2a", "Paclitaxel (nab)", "Filgrastim", "Leuprorelin; Tamoxifen"])

# * add a column with recoded substances
results = rec.add_substance(
    col_with_substances=fake_data,
    col_with_ref_substances=reference_list["substanz"],
    threshold=0.85
)

Options and parameters

threshold

The function features some parameters: The threshold parameter defines the accuracy. The lower the more matches but higher values lead to more accurate matches. The graph below plots the number of substances extracted from free text fields against the corresponding threshold parameter. There is a tradeoff between the number of matches and their accuracy. A threshold value of 0.85 is set as default as it usually ensures sufficient accuracy.

only_first_match

The option only_first_match = True should be used if the user wants to allow only one match per free text field. Even if there are several substances in the free text field such as "Leuprorelin; Tamoxifen", the function will return only the first match. If the option is set to "False", the function can return multiple hits. For the input "Interferon alpha-2a weekly i.v.", the function might return two substances "Interferon alpha-2a" and "Interferon alpha-2b" (assuming they are both on the reference list). Results based on both options is shown below.

Credits

Thanks to @smeisegeier for helpful feedback on the code and for making the function available as a python package.