Skip to main content

ideadensity

PyPI - Version Unit Tests

Python library for computing propositional idea density.

Table of Contents

Introduction

ideadensity is a Python library which determines the propositional idea density of an English text automatically. This project aims to make this functionality more accessible to Python developers and researchers. ideadensity provides two ways of computing idea density:

  • CPIDR. The CPIDR implementation in ideadensity is a direct port of the Computerized Propositional Idea Density Rater (CPIDR) 3.2 (Brown et al., 2008) [1]
  • DEPID. This library implements the DEPID algorithm described by Sirts et al (2017) [2]

Here's a quick example of how to use ideadensity:

from ideadensity import cpidr, depid

text = "The quick brown fox jumps over the lazy dog."
cpidr_word_count, proposition_count, cpidr_density, word_list = cpidr(text)
depid_density, depid_word_count, dependencies = depid(text)

print(f"CPIDR density: {cpidr_density:.3f}")
print(f"DEPID density: {depid_density:.3f}")

What is Idea Density?

Idea density, also known as propositional density, is a measure of the amount of information conveyed relative to the number of words used. It's calculated by dividing the number of expressed propositions by the number of words. This metric has applications in various fields, including linguistics, cognitive science, and healthcare research.

Installation

Using pip

  1. Install the package
pip install ideadensity
  1. Download the required spaCy model:
python -m spacy download en_core_web_sm

Using poetry

poetry add ideadensity
python -m spacy download en_core_web_sm

Python 3.10+ is supported, including 3.13.

The desktop GUI is an optional extra (the plain library no longer installs PyQt6):

pip install "ideadensity[gui]"

Usage

CPIDR

Here's a simple example of how to use CPIDR:

from ideadensity import cpidr

text = "The quick brown fox jumps over the lazy dog."
word_count, proposition_count, density, word_list = cpidr(text)

print(f"Word count: {word_count}")
print(f"Proposition count: {proposition_count}")
print(f"Idea density: {density:.3f}")

# Analyzing speech
speech_text = "Um, you know, I think that, like, the weather is nice today."
word_count, proposition_count, density, word_list = cpidr(speech_text, speech_mode=True)

print(f"Speech mode - Idea density: {density:.3f}")

# Detailed word analysis
for word in word_list.items:
    if word.is_word:
        print(f"Token: {word.token}, Tag: {word.tag}, Is proposition: {word.is_proposition}")

Speech Mode

ideadensity CPIDR mode supports a speech mode that handles common speech patterns and fillers differently from written text. When analyzing transcripts or spoken language, use the speech_mode=True parameter for more accurate results.

DEPID

Here's an example of how to use the DEPID functionality:

from ideadensity import depid

text = "The quick brown fox jumps over the lazy dog."
density, word_count, dependencies = depid(text)
print(f"Word count: {word_count}")
print(f"Idea density: {density:.3f}")
print("Dependencies:")
for dep in dependencies:
    print(f"Token: {dep[0]}, Dependency: {dep[1]}, Head: {dep[2]}")

DEPID-R

DEPID-R counts distinct dependencies.

from ideadensity import depid

text = "This is a test of DEPID-R. This is a test of DEPID-R"
density, word_count, dependencies = depid(text, is_depid_r=True)

print(f"DEPID-R idea density: {density:.3f}")

Using custom filters

ideadensity DEPID mode supports custom filtering of sentences and tokens. By default, ideadensity uses filters described by (Sirts et al., 2017):

  • Sentence filter.
    • Filter out sentences with "I" or "You" as the subject of the sentence (i.e. if the "I" or "You" token dependency is "nsubj" and it's head dependency is the root).
    • Note: Sirts et al (2017) also filters out vague sentences using SpeciTeller. That is a filter which ideadensity does not yet implement.
  • Token filters:
    • Filter out "det" dependencies if the token is "a", "an" or "the".
    • Filter out "nsubj" dependencies if the token is "it" or "this".
    • Filter out all "cc" dependencies.

This example demonstrates how to apply your own custom filters to modify the analysis. The sentence_filters and token_filters parameters allow you to customize the DEPID algorithm to suit your specific needs.

def custom_sentence_filter(sent):
    return len(sent) > 3
def custom_token_filter(token):
    return token.pos_ != "DET"
text_with_filters = "I run. The quick brown fox jumps over the lazy dog."
density, word_count, dependencies = depid(text_with_filters,
sentence_filters=[custom_sentence_filter],
token_filters=[custom_token_filter])
print(f"\nWith custom filters - Idea density: {density:.3f}")

Command Line Interface

The package includes a command line interface for quick analysis of text: Command line options:

  • --text TEXT: Directly provide text for analysis (can include multiple words)
  • --file FILE: Path to a file containing text to analyze
  • --speech-mode: Enable speech mode for analyzing transcripts (filters common fillers)
  • --csv CSV: Export token details to a CSV file at the specified path
  • --txt TXT: Export results to a TXT file in CPIDR format at the specified path

Note: You must provide either --text or --file when using the command line interface.

# Analyze text directly from command line
python main.py --text "The quick brown fox jumps over the lazy dog."

# Analyze text from a file
python main.py --file sample.txt

# Use speech mode with text from a file
python main.py --file transcript.txt --speech-mode

# Export token details to a CSV file
python main.py --text "This is a test sentence." --csv output.csv

# Export results in CPIDR-compatible format to a TXT file
python main.py --text "This is a test sentence." --txt output.txt

# Export in both formats
python main.py --file sample.txt --csv output.csv --txt output.txt

Graphical User Interface

Use one of the provided downloads for your operating system, or clone this repository and run:

python main.py

Export Formats

CSV Export: The CSV export includes detailed information about each token with the following columns:

  • Token: The actual word or token
  • Tag: The part-of-speech tag
  • Is Word: Whether the token is considered a word (True/False)
  • Is Proposition: Whether the token is considered a proposition (True/False)
  • Rule Number: The rule number that identified the token as a proposition (if applicable)

TXT Export: The TXT export produces a file in a format compatible with the original CPIDR tool:

ideadensity 0.2.11

"This is a test sentence...."
 054 PRP  W   This
 200 VBZ  W P is
 201 DT   W   a
 200 JJ   W P test
     NN   W   sentence
     .        .

     2 propositions
     5 words
 0.400 density

Each line in the token section includes:

  • Rule number (if available)
  • Part-of-speech tag
  • Word marker (W if the token is a word)
  • Proposition marker (P if the token is a proposition)
  • The token text

Requirements

  • Python 3.10+
  • spaCy 3.7.5+

Development Setup

To set up the development environment:

  1. Clone the repository
  2. Install Poetry if you haven't already: pip install poetry
  3. Install project dependencies: poetry install
  4. Install the required spaCy model: poetry run python -m spacy download en_core_web_sm
  5. Activate the virtual environment: poetry shell

Running Tests

To run the tests, use pytest:

pytest tests/

CPIDR Parity with CPIDR 3.2

Because this port uses spaCy as a part-of-speech tagger instead of the original program's MontyLingua, there is a very slight difference in the reported idea density. This port includes unit tests containing 847 words of text. ideadensity: 434 propositions. 0.512 idea density CPIDR 3.2: 436 propositions. 0.515 idea density

For more information about the original CPIDR 3.2, please visit CASPR's official page.

References

[1] Brown, C., Snodgrass, T., Kemper, S. J., Herman, R., & Covington, M. A. (2008). Automatic measurement of propositional idea density from part-of-speech tagging. Behavior Research Methods, 40(2), 540-545. https://doi.org/10.3758/BRM.40.2.540

[2] Sirts, K., Piguet, O., & Johnson, M. (2017). Idea density for predicting Alzheimer's disease from transcribed speech. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017) (pp. 322-332). Association for Computational Linguistics. https://doi.org/10.18653/v1/K17-1033

Citing

If you use ideadensity in your research, please cite both the tool and the paper for the method you used:

  • Always cite the tool (see CITATION.cff, or use GitHub's "Cite this repository" button):

    Robison, J. (2026). ideadensity (Version 0.6.2) [Computer software]. https://github.com/jrrobison1/ideadensity

  • If you used cpidr, also cite the original CPIDR paper, of which this is a port: Brown et al. (2008) [1].

  • If you used depid, also cite the paper that introduced the DEPID algorithm: Sirts et al. (2017) [2].

For reproducibility, report the ideadensity version, the spaCy version, and the spaCy model version — idea-density scores depend on the tagger and parser, so the same text can score differently across spaCy releases. The library reports all three:

import ideadensity
print(ideadensity.reproducibility_info())
# {'ideadensity': '0.6.0', 'spacy': '3.8.11',
#  'spacy_model': 'en_core_web_sm', 'spacy_model_version': '3.8.0'}

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

  1. Fork the repository
  2. Create your feature branch (git checkout -b feature/AmazingFeature)
  3. Commit your changes (git commit -m 'Add some AmazingFeature')
  4. Push to the branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

Please ensure that your code passes all tests and follows the project's coding style.

License

This project is licensed under the GNU General Public License v2.0. See the LICENSE file for details.

ideadensity's CPIDR implementation is a port of the original CPIDR 3.2, which was released under GPL v2. This project maintains the same license to comply with the terms of the original software.

Metadata

Release files for ideadensity 0.6.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ideadensity 0.6.3
File Size Uploaded Provenance Actions
ideadensity-0.6.3.tar.gz 32.0 kB Without provenance Details for ideadensity-0.6.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ideadensity 0.6.3
File Interpreter ABI Platform Actions
ideadensity-0.6.3-py3-none-any.whl Python 3 none any Details for ideadensity-0.6.3-py3-none-any.whl

Total release size: 63.9 kB

Release files / ideadensity-0.6.3.tar.gz

Download URL ideadensity-0.6.3.tar.gz
Size 32.0 kB
Tags Source
SHA-256 checksum
How to use checksums
a4b5c2807b90df37a301d184753732a2e6cbef1eb937d4520870d66142335e5b
BLAKE2b-256 checksum
How to use checksums
cdec707e71632ed5d2a49061ce89f4b32eabc9ea5132d3d4e881a05eb26733b4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Release files / ideadensity-0.6.3-py3-none-any.whl

Download URL ideadensity-0.6.3-py3-none-any.whl
Size 31.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a6a7b2fdb1eeb8e55419495c98d8bd7dcd62ceb2a15573a70c561c2a0648c656
BLAKE2b-256 checksum
How to use checksums
3ff046f9fa394761cd330f57c0da38cdf7d584be01ceb2db67afaf95615875a9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Security for ideadensity 0.6.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page