medspellchecker 0.0.4

Fast and effective spellchecker for Russian medical texts

MedSpellChecker

Fast and effective tool for correcting spelling errors in Russian medical texts. The tool takes the raw medical text and returns the corrected text in lemmatized form.

This project is in its final stages. If you have found a bug or would like to improve something, please create an issue or pull request.

How to install

The project is written in Python 3.9 and works on Python version 3.9 or higher.

• From the pip repository

  pip install medspellchecker
  

• Or from the sources

  • Install requirements

    pip install -r requirements.txt
    

  • Install the package

    pip install .
    

Note! MedSpellChecker has the editdistpy package as a dependency. To install it you will need Microsoft Visual C++ 14 (for Windows) or build-essential package (for Linux).

How to use

An example of using the `medspellchecker' is shown below.

from medspellchecker.tool.medspellchecker import MedSpellchecker
from medspellchecker.tool.distilbert_candidate_ranker import RuDistilBertCandidateRanker

candidate_ranker = RuDistilBertCandidateRanker()
spellchecker = MedSpellchecker(candidate_ranker)
fixed_text = spellchecker.fix_text(
    "У больногодиагностирован инфркт и тубркулез"
)

print(fixed_text)
# -> "У больной диагностировать инфаркт и тубркулез"

1. The first two lines import the main class of the package.
2. Lines 3 and 4 then import one of the three available classes to rank the edit candidates.
3. Line 6 creates an instance of the candidate ranking class based on the fine-tuned DistilBert model.
4. Line 7 creates an instance of the class to correct spelling errors.
5. On line 8, the fix_text method is called, which takes the raw text and returns the corrected text.
6. Finally line 10 prints the corrected result, which looks like "У больной диагностировать инфаркт и тубркулез".
7. In this way, the package can be used to correct spelling errors in Russian medical texts in a few lines.

Tips for using the tool

1. Initiate and create MedSpellchecker class once. And then use it many times to correct Russian medical texts.
2. The tool must be used before all the preprocessing steps. This way the various preprocessing steps cannot affect the result of the tool correction.
3. Nevertheless, the input raw text must be cleared of all sorts of tags, technical symbols and other trash. The input text should be plain text without any structure.
4. Before using the tool on the whole dataset, test how it works on a small chunk to choose the best available candidate ranker.
5. If the available candidate rankers do not perform well enough, consider fine-tuning the BERT-based model on your dataset and using the candidate ranker on the fine-tuned model. The quality of the corrections should improve.

Internals

MedSpellChecker uses the SymDel algorithm to speed up the generation of correction candidates, and a fine-tuned BERT-based machine learning model to rank candidates and select the best fit.

This architecture allows each component to be developed almost independently and the correction process to be implemented flexibly.

• Spellchecker Manager - responsible for coordinating other components and implementing high-level logic.
• Preprocessor and PostProcessor - responsible for splitting the incoming text and assembling the result.
• Dictionary - contains a dictionary of correct words, which allows to check the correct word or not.
• Edit Distance Index - allows to optimize and speed up the calculation of the editing distance required to generate candicates for fixing an incorrect word.
• Error Model - responsible for generating candidates for fixing incorrect words.
• Language Model - based on the fine-tuned BERT-based model, ranks candidates for fixing and selects the most suitable word for correction.

Supported Candidate Rankers

Candidate rankers rank the candidates to replace the incorrect word and choose the most appropriate one.

The tool contains several rankers based on different BERT-based models. However, the list of supported rankers can easily be extended with rankers based on any other approach and models. It is sufficient to implement the AbstractCandidateRanker interface.

Available rankers out of the box:

• RuRobertaCandidateRanker based on DmitryPogrebnoy/MedRuRobertaLarge model
• RuDistilBertCandidateRanker based on DmitryPogrebnoy/MedDistilBertBaseRuCased model
• RuBertTiny2CandidateRanker based on DmitryPogrebnoy/MedRuBertTiny2 model
• RuBioRobertCandidateRanker based on alexyalunin/RuBioRoBERTa model
• RuBioBertCandidateRanker based on alexyalunin/RuBioBERT model

More information

This project is part of master's thesis. The current state is the result of the first year of work. More details about MedSpellCHecker you can find in the text of the term report .