Skip to main content

Calculate statistical features from text

Project description

textstat

Python package to calculate statistics from text to determine readability, complexity and grade level of a particular corpus.

Downloads

Usage

>>> import textstat

>>> test_data = (
    "Playing games has always been thought to be important to "
    "the development of well-balanced and creative children; "
    "however, what part, if any, they should play in the lives "
    "of adults has never been researched that deeply. I believe "
    "that playing games is every bit as important for adults "
    "as for children. Not only is taking time out to play games "
    "with our children and other adults valuable to building "
    "interpersonal relationships but is also a wonderful way "
    "to release built up tension."
)

>>> textstat.flesch_reading_ease(test_data)
>>> textstat.smog_index(test_data)
>>> textstat.flesch_kincaid_grade(test_data)
>>> textstat.coleman_liau_index(test_data)
>>> textstat.automated_readability_index(test_data)
>>> textstat.dale_chall_readability_score(test_data)
>>> textstat.difficult_words(test_data)
>>> textstat.linsear_write_formula(test_data)
>>> textstat.gunning_fog(test_data)
>>> textstat.text_standard(test_data)

The argument (text) for all the defined functions remains the same - i.e the text for which statistics need to be calculated.

Install

You can install textstat either via the Python Package Index (PyPI) or from source.

Install using pip

pip install textstat

Install using easy_install

easy_install textstat

Install lastest version from GitHub

git clone https://github.com/shivam5992/textstat.git
cd textstat
pip install .

Install from PyPI

Download the latest version of textstat from http://pypi.python.org/pypi/textstat/

You can install it by doing the following:

tar xfz textstat-*.tar.gz
cd textstat-*/
python setup.py build
python setup.py install # as root

List of Functions

Syllable Count

textstat.syllable_count(text, lang='en_US')

Returns the number of syllables present in the given text.

Uses the Python module Pyphen for syllable calculation. Optional lang specifies to Pyphen which language dictionary to use.

Default is 'en_US', 'en_GB' will also work.

Lexicon Count

textstat.lexicon_count(text, removepunct=True)

Calculates the number of words present in the text. Optional removepunct specifies whether we need to take punctuation symbols into account while counting lexicons. Default value is True, which removes the punctuation before counting lexicon items.

Sentence Count

textstat.sentence_count(text)

Returns the number of sentences present in the given text.

The Flesch Reading Ease formula

textstat.flesch_reading_ease(text)

Returns the Flesch Reading Ease Score.

The following table can be helpful to assess the ease of readability in a document.

The table is an example of values. While the maximum score is 121.22, there is no limit on how low the score can be. A negative score is valid.

Score Difficulty
90-100 Very Easy
80-89 Easy
70-79 Fairly Easy
60-69 Standard
50-59 Fairly Difficult
30-49 Difficult
0-29 Very Confusing

Further reading on Wikipedia

The Flesch-Kincaid Grade Level

textstat.flesch_kincaid_grade(text)

Returns the Flesch-Kincaid Grade of the given text. This is a grade formula in that a score of 9.3 means that a ninth grader would be able to read the document.

Further reading on Wikipedia

The Fog Scale (Gunning FOG Formula)

textstat.gunning_fog(text)

Returns the FOG index of the given text. This is a grade formula in that a score of 9.3 means that a ninth grader would be able to read the document.

Further reading on Wikipedia

The SMOG Index

textstat.smog_index(text)

Returns the SMOG index of the given text. This is a grade formula in that a score of 9.3 means that a ninth grader would be able to read the document.

Texts of fewer than 30 sentences are statistically invalid, because the SMOG formula was normed on 30-sentence samples. textstat requires atleast 3 sentences for a result.

Further reading on Wikipedia

Automated Readability Index

textstat.automated_readability_index(text)

Returns the ARI (Automated Readability Index) which outputs a number that approximates the grade level needed to comprehend the text.

For example if the ARI is 6.5, then the grade level to comprehend the text is 6th to 7th grade.

Further reading on Wikipedia

The Coleman-Liau Index

textstat.coleman_liau_index(text)

Returns the grade level of the text using the Coleman-Liau Formula. This is a grade formula in that a score of 9.3 means that a ninth grader would be able to read the document.

Further reading on Wikipedia

Linsear Write Formula

textstat.linsear_write_formula(text)

Returns the grade level using the Linsear Write Formula. This is a grade formula in that a score of 9.3 means that a ninth grader would be able to read the document.

Further reading on Wikipedia

Dale-Chall Readability Score

textstat.dale_chall_readability_score(text)

Different from other tests, since it uses a lookup table of the most commonly used 3000 English words. Thus it returns the grade level using the New Dale-Chall Formula.

Score Understood by
4.9 or lower average 4th-grade student or lower
5.0–5.9 average 5th or 6th-grade student
6.0–6.9 average 7th or 8th-grade student
7.0–7.9 average 9th or 10th-grade student
8.0–8.9 average 11th or 12th-grade student
9.0–9.9 average 13th to 15th-grade (college) student

Further reading on Wikipedia

Readability Consensus based upon all the above tests

textstat.text_standard(text, float_output=False)

Based upon all the above tests, returns the estimated school grade level required to understand the text.

Optional float_output allows the score to be returned as a float. Defaults to False.

Contributing

If you find any problems, you should open an issue.

If you can fix an issue you've found, or another issue, you should open a pull request.

  1. Fork this repository on GitHub to start making your changes to the master branch (or branch off of it).
  2. Write a test which shows that the bug was fixed or that the feature works as expected.
  3. Send a pull request!

Development setup

It is recommended you use a virtual environment, or Pipenv to keep your development work isolated from your systems Python installation.

$ git clone https://github.com/<yourname>/textstat.git  # Clone the repo from your fork
$ cd textstat
$ pip install -r requirements.txt  # Install all dependencies

$ # Make changes

$ python -m unittest test.py  # Run tests

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

textstat-0.5.6.tar.gz (19.2 kB view details)

Uploaded Source

Built Distributions

textstat-0.5.6-py3-none-any.whl (16.2 kB view details)

Uploaded Python 3

textstat-0.5.6-py2-none-any.whl (16.3 kB view details)

Uploaded Python 2

File details

Details for the file textstat-0.5.6.tar.gz.

File metadata

  • Download URL: textstat-0.5.6.tar.gz
  • Upload date:
  • Size: 19.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.18.4 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.6.7

File hashes

Hashes for textstat-0.5.6.tar.gz
Algorithm Hash digest
SHA256 5e1342bf87b4660f5437a36ce0a12cc987885187527c97c6b1f19557811df4d6
MD5 a7c0aa86b6e5e4032ba353f41516d0d2
BLAKE2b-256 545cbb959d97c28f321483dc37da8b3cb85ba9a3904deba967f917b7dc401121

See more details on using hashes here.

File details

Details for the file textstat-0.5.6-py3-none-any.whl.

File metadata

  • Download URL: textstat-0.5.6-py3-none-any.whl
  • Upload date:
  • Size: 16.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.18.4 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.6.7

File hashes

Hashes for textstat-0.5.6-py3-none-any.whl
Algorithm Hash digest
SHA256 fd225f95cb558fa2923b2bea4991f77e8dcd66eb9b544824a297b04a6a0d4425
MD5 a4d67cb24ff9fb78b4d347ac6c2626f6
BLAKE2b-256 667397bb64c89d6f2b24be6ad76007823e19b4c32ed4d484420b3ec6892ac440

See more details on using hashes here.

File details

Details for the file textstat-0.5.6-py2-none-any.whl.

File metadata

  • Download URL: textstat-0.5.6-py2-none-any.whl
  • Upload date:
  • Size: 16.3 kB
  • Tags: Python 2
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.18.4 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.6.7

File hashes

Hashes for textstat-0.5.6-py2-none-any.whl
Algorithm Hash digest
SHA256 c50ad2691763c74508e35e554da2ad8aee748537c999388f93e218fcab9ab12f
MD5 ee9f9ee64f382ba494b28f047e730fd3
BLAKE2b-256 8cd9ca110b1e5e410ebe1d4cae0f082c15a7b2e4cfcd9c1c07eedd6a499c6013

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page