Duplicate Text Finder
duptextfinder is a python library to detect duplicated zones in text. Primarily meant to detect
copy/paste across medical documents. Should be faster than python's built-in
difflib algorithm and more robust to whitespace, newlines and other irrelevant
characters.
Installation
duptextfinder can be installed through pip:
pip install duptextfinder
Usage
from pathlib import Path
from duptextfinder import CharFingerprintBuilder, DuplicateFinder
# load some text files
texts = [p.read_text() for p in Path("some/dir").glob("*.txt")]
# init fingerprint and duplicate finder
fingerprintBuilder = CharFingerprintBuilder(fingerprintLength=15)
duplicateFinder = DuplicateFinder(fingerprintBuilder, minDuplicateLength=15)
# call findDuplicates() on each file
for i, text in enumerate(texts):
id = f"D{i}"
duplicates = duplicateFinder.findDuplicates(id, text)
for duplicate in duplicates:
print(
f"sourceDoc={duplicate.sourceDocId}, "
f"sourceStart={duplicate.sourceSpan.start}, "
f"sourceEnd={duplicate.sourceSpan.end}, "
f"targetStart={duplicate.targetSpan.start}, "
f"targetEnd={duplicate.targetSpan.end}"
)
duplicated_text = text[duplicate.targetSpan.start : duplicate.targetSpan.end]
print(duplicated_text)
WordFingerprintBuilder can be used instead of CharFingerprintBuilder. For
more details, refer to the docstrings of DuplicateFinder,
CharFingerprintBuilder and WordFingerprintBuilder.
How to run tests
- Install package in editable mode with test and extra dependencies by running
pip install -e ".[tests, ncls, intervaltree]"in the repo directory - Launch
pytest tests/
About ncls and intervaltree
This tool can be used without any additional dependencies, but performance can be improved when using interval trees. To benefit from this you well need to install either the ncls package or the intervaltree package.
References
- Evaluating the Impact of Text Duplications on a Corpus of More than 600,000 Clinical Narratives in a French Hospital. https://www.hal.inserm.fr/hal-02265124/
Metadata
Release files for duptextfinder 0.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| duptextfinder-0.3.0.tar.gz | 19.4 kB | Without provenance | Details for duptextfinder-0.3.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| duptextfinder-0.3.0-py3-none-any.whl | Python 3 | none | any | Details for duptextfinder-0.3.0-py3-none-any.whl |
Total release size: 36.6 kB
Release files / duptextfinder-0.3.0.tar.gz
| Download URL | duptextfinder-0.3.0.tar.gz |
|---|---|
| Size | 19.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a8ff3f3128bdc56157b2d09778e9ca2d73b093fea5419947418c07f83eaba08e
|
|
BLAKE2b-256 checksum How to use checksums |
67668796a3b0156aa70584e768bd66002a219eec868c215b9c83a90e28d26c5b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/4.0.2 CPython/3.11.7
|
Release files / duptextfinder-0.3.0-py3-none-any.whl
| Download URL | duptextfinder-0.3.0-py3-none-any.whl |
|---|---|
| Size | 17.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b23072f5839a71240c43e288902024ee09a4491a64eb102619e4028f0253e37d
|
|
BLAKE2b-256 checksum How to use checksums |
c650c45b26f67e3301efeb9cfd11b7509f18c794439cbfffe250392380f11e1d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/4.0.2 CPython/3.11.7
|
Security for duptextfinder 0.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.