deduce 3.0.3

Deduce: de-identification method for Dutch medical text

deduce

Deduce 3.0.0 is out! It is way more accurate, and faster too. It's fully backward compatible, but some functionality is scheduled for removal, read more about it here: docs/migrating-to-v3

• :sparkles: Remove sensitive information from clinical text written in Dutch
• :mag: Rule based logic for detecting e.g. names, locations, institutions, identifiers, phone numbers
• :triangular_ruler: Useful out of the box, but customization higly recommended
• :seedling: Originally validated in Menger et al. (2017), but further optimized since

:exclamation: Deduce is useful out of the box, but please validate and customize on your own data before using it in a critical environment. Remember that de-identification is almost never perfect, and that clinical text often contains other specific details that can link it to a specific person. Be aware that de-identification should primarily be viewed as a way to mitigate risk of identification, rather than a way to obtain anonymous data.

Currently, deduce can remove the following types of Protected Health Information (PHI):

• :bust_in_silhouette: person names, including prefixes and initials
• :earth_americas: geographical locations smaller than a country
• :hospital: names of hospitals and healthcare institutions
• :calendar: dates (combinations of day, month and year)
• :birthday: ages
• :1234: BSN numbers
• :1234: identifiers (7+ digits without a specific format, e.g. patient identifiers, AGB, BIG)
• :phone: phone numbers
• :e-mail: e-mail addresses
• :link: URLs

Citing

If you use deduce, please cite the following paper:

Menger, V.J., Scheepers, F., van Wijk, L.M., Spruit, M. (2017). DEDUCE: A pattern matching method for automatic de-identification of Dutch medical text, Telematics and Informatics, 2017, ISSN 0736-5853

Installation

pip install deduce

Getting started

The basic way to use deduce, is to pass text to the deidentify method of a Deduce object:

from deduce import Deduce

deduce = Deduce()

text = (
    "betreft: Jan Jansen, bsn 111222333, patnr 000334433. De patient J. Jansen is 64 jaar oud en woonachtig in "
    "Utrecht. Hij werd op 10 oktober 2018 door arts Peter de Visser ontslagen van de kliniek van het UMCU. "
    "Voor nazorg kan hij worden bereikt via [e-mail] of [phone]."
)

doc = deduce.deidentify(text)

The output is available in the Document object:

from pprint import pprint

pprint(doc.annotations)

AnnotationSet({
    Annotation(text="[phone]", start_char=272, end_char=284, tag="telefoonnummer"),
    Annotation(text="111222333", start_char=25, end_char=34, tag="bsn"),
    Annotation(text="Peter de Visser", start_char=153, end_char=168, tag="persoon"),
    Annotation(text="[e-mail]", start_char=247, end_char=268, tag="email"),
    Annotation(text="patient J. Jansen", start_char=56, end_char=73, tag="patient"),
    Annotation(text="Jan Jansen", start_char=9, end_char=19, tag="patient"),
    Annotation(text="10 oktober 2018", start_char=127, end_char=142, tag="datum"),
    Annotation(text="64", start_char=77, end_char=79, tag="leeftijd"),
    Annotation(text="000334433", start_char=42, end_char=51, tag="id"),
    Annotation(text="Utrecht", start_char=106, end_char=113, tag="locatie"),
    Annotation(text="UMCU", start_char=202, end_char=206, tag="instelling"),
})

print(doc.deidentified_text)

"""betreft: [PERSOON-1], bsn [BSN-1], patnr [ID-1]. De [PERSOON-1] is [LEEFTIJD-1] jaar oud en woonachtig in 
[LOCATIE-1]. Hij werd op [DATUM-1] door arts [PERSOON-2] ontslagen van de kliniek van het [INSTELLING-1]. 
Voor nazorg kan hij worden bereikt via [EMAIL-1] of [TELEFOONNUMMER-1]."""

Additionally, if the names of the patient are known, they may be added as metadata, where they will be picked up by deduce:

from deduce.person import Person

patient = Person(first_names=["Jan"], initials="JJ", surname="Jansen")
doc = deduce.deidentify(text, metadata={'patient': patient})

print (doc.deidentified_text)

"""betreft: [PATIENT], bsn [BSN-1], patnr [ID-1]. De [PATIENT] is [LEEFTIJD-1] jaar oud en woonachtig in 
[LOCATIE-1]. Hij werd op [DATUM-1] door arts [PERSOON-2] ontslagen van de kliniek van het [INSTELLING-1]. 
Voor nazorg kan hij worden bereikt via [EMAIL-1] of [TELEFOONNUMMER-1]."""

As you can see, adding known names keeps references to [PATIENT] in text. It also increases recall, as not all known names are contained in the lookup lists.

Versions

For most cases the latest version is suitable, but some specific milestones are:

• 3.0.0 - Many optimizations in accuracy, smaller refactors, further speedups
• 2.0.0 - Major refactor, with speedups, many new options for customizing, functionally very similar to original
• 1.0.8 - Small bugfixes compared to original release
• 1.0.1 - Original release with Menger et al. (2017)

Detailed versioning information is accessible in the changelog.

Documentation

All documentation, including a more extensive tutorial on using, configuring and modifying deduce, and its API, is available at: docs/tutorial

Contributing

For setting up the dev environment and contributing guidelines, see: docs/contributing

Authors

• Vincent Menger - Initial work
• Jonathan de Bruin - Code review
• Pablo Mosteiro - Bug fixes, structured annotations

License

This project is licensed under the GNU General Public License v3.0 - see the LICENSE.md file for details