Skip to main content
Join the official 2020 Python Developers SurveyStart the survey!

Language model powered proof reader for correcting contextual errors in natural language.

Project description

lmproof - Language Model Proof Reader

Library to do proof-reading corrections for Grammatical Errors, spelling errors, confused word errors and other errors using pre-trained Language Models.

Usage

Install spacy model by python -m spacy download en. Then try out this snippet.

import lmproof
proof_reader = lmproof.load("en")
source = "The foxes living on the Shire is brown.'"
corrected = proof_reader.proofread(source) # "The foxes living in the Shire are brown."

How it works?

We use the language model based scoring approach mentioned in Christopher Bryant and Ted Briscoe. 2018 with few changes.

Unlike many approaches to GEC, this approach does NOT require annotated training data and mainly depends on a monolingual language model. The program works by iteratively comparing certain words in a text against alternative candidates and applying a correction if one of these candidates is more probable than the original word. These correction candidates are variously generated by a word inflection library or are otherwise defined manually. Currently, this system only corrects:

Non-words (e.g. freind and informations)
Morphology (e.g. eat, ate, eaten, eating, etc.)
Common Determiners and Prepositions (e.g. the, a, in, at, to, etc.)
Commonly Confused Words (e.g. bear/bare, lose/loose, etc.)

This work builds upon https://github.com/chrisjbryant/lmgec-lite/

Components

Language Models

Inflection generators

  • LemmInflect is used to lemmatize and generate inflections for candidate proposals to the language model.

Spell Checker

  • symspellpy is used for obtaining spell check candidates.

The components are highly modularised to facilitate experimentation with newer scorers and support more languages. Pre-trained language models for other languages, inflectors, common error patterns can be easily added to support more languages.

TODOs

  • Use edits in existing GEC corpus to generate candidates.
  • Tests
  • Publish benchmarks of the model.
  • Think of simple ways to generate insertion candidates.
  • Add more languages.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for lmproof, version 0.3.0
Filename, size File type Python version Upload date Hashes
Filename, size lmproof-0.3.0-py3-none-any.whl (639.8 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size lmproof-0.3.0.tar.gz (640.5 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page