smoothtext 0.3.2

A Python library for text readability analysis, supporting multiple languages.

SmoothText

---

---

Introduction

SmoothText is a Python library for calculating readability scores of texts and statistical information for texts in multiple languages.

The design principle of this library is to ensure high accuracy.

Requirements

Python 3.10 or higher.

External Dependencies

Library	Version	License	Notes
NLTK	>=3.9.1	Apache 2.0	Conditionally optional.
Stanza	>=1.10.1	Apache 2.0	Conditionally optional.
CMUdict	>=1.0.32	GPLv3+	Required if Stanza is the selected backend.
Unidecode	>=1.3.8	GNU GPLv2	Required.
Pyphen	>=0.17.0	GPL 2.0+/LGPL 2.1+/MPL 1.1	Required.
emoji	>=2.14.1	BSD	Required.

Either NLTK or Stanza must be installed and used with the SmoothText library.

Features

Readability Analysis

SmoothText can calculate readability scores of text in the following languages, using the following formulas.

Formula/Language	English	German	Turkish
Automated Readability Index	✔	❌	❌
Flesch Reading Ease	✔	✔	✔ Ateşman
Flesch-Kincaid Grade	✔	✔ Wiener Sachtextformel	✔ Bezirci-Yılmaz
Flesch-Kincaid Grade Simplified	✔	❌	❌

Notes:

English:

• Formulas work best with US English. However, SmoothText supports both US English and GB English.

German:

• Flesch Reading Ease is applicable to German texts. SmoothText handles the language-specific adaptations of the formula.
• Wiener Sachtextformel is the German adaptation of Flesch-Kincaid Grade.

Turkish:

• Ateşman is the Turkish adaptation of Flesch Reading Ease.
• Bezirci-Yılmaz is the Turkish adaptation of Flesch-Kincaid Grade.

Sentencizing, Tokenizing, and Syllabifying

SmoothText can extract sentences, words, or syllables from texts.

Reading Time

SmoothText can calculate how long would a text take to read.

Counting Frequency of Words

SmoothText can count the frequency of (lemmatized) words.

Installation

You can install SmoothText via pip.

pip
install
smoothtext

Usage

Importing and Initializing the Library

SmoothText comes with four submodules: Backend, Language, ReadabilityFormula and SmoothText.

from smoothtext import Backend, Language, ReadabilityFormula, SmoothText

Instancing

SmoothText was not designed to be used with static methods. Thus, an instance must be created to access its methods.

When creating an instance, the language and the backend to be used with it can be specified.

The following will create a new SmoothText instance configured to be used with the English language (by default, the United States variant) using NLTK as the backend.

st = SmoothText('en', 'nltk')

Once an instance is created, its backend cannot be changed, but its working language can be changed at any time.

st.language = 'tr'  # Now configured to work with Turkish.
st.language = 'en-gb'  # Switching back to English, but to the United Kingdom variant.

Readying the Backends

When an instance is created, the instance will first attempt to import and download the required backend/language data. To avoid this, and to prepare the required packages in advance, we can use the static SmoothText.prepare() method.

SmoothText.prepare('nltk', 'en,tr')  # Preparing NLTK to be used with English and Turkish

Computing Readability Scores

Each language has its own set of readability formulas. When computing the readability score of a text in a language, one of the supporting formulas must be used. Using SmoothText, there are three ways to perform this calculation.

text: str = 'Forrest Gump is a 1994 American comedy-drama film directed by Robert Zemeckis.'  # https://en.wikipedia.org/wiki/Forrest_Gump

# Generic computation method
st.compute_readability(text, ReadabilityFormula.Flesch_Reading_Ease)

# Using instance as a callable for generic computation
st(text, ReadabilityFormula.Flesch_Reading_Ease)

# Specific formula method
st.flesch_reading_ease(text)

Tokenizing and Calculating Text Statistics

SmoothText is designed to work with sentences, words/tokens, and syllables.

text = 'This is a test sentence. This is another test sentence. This is a third test sentence.'

st.count_sentences(text)
# Output: 3

st.count_words(text)
# Output: 14

st.count_syllables(text)
# Output: 21

Other Features

Refer to the documentation for a complete list of available methods.

Inconsistencies

Backend Related Inconsistencies

• NLTK and Stanza have different tokenization rules. This may cause differences in the number of tokens/sentences between the two backends.

Language Related Inconsistencies

• The syllabification of words may differ within the same language variant. For example, the word "hello" has two syllables in American English but one in British English.

Documentation

See here for API documentation.

License

SmoothText has an MIT license. See LICENSE.