ContentCoder

ContentCoder is a Python-based text analysis tool that enables users to process and analyze text using custom linguistic dictionaries. It is inspired by tools like LIWC (Linguistic Inquiry and Word Count) and provides robust methods for tokenization, text analysis, and frequency calculations.

Note: Approximately 98% of this README was generated by ChatGPT — it may not be entirely accurate, but at a quick glance, it looks pretty spot-on.

Features

• Custom Dictionary-Based Analysis
• Support for LIWC-style dictionaries (2007 & 2022 formats)
• Efficient text tokenization
• Wildcard and abbreviation handling
• Punctuation and big word analysis
• Dictionary export in multiple formats (JSON, CSV, Poster format, etc.)
• High-performance wildcard matching with memory optimization

---

Installation

Ensure you have Python 3.9+ installed. ContentCoder is all native Python and does not require dependencies for installation.

pip install contentcoder

---

Folder Structure

src/contentcoder/
│── __init__.py
│─ ContentCoder.py
│─ ContentCodingDictionary.py
│─ happiestfuntokenizing.py
│─ create_export_dir.py

---

Quick Start

1. Import the ContentCoder class

from contentcoder.ContentCoder import ContentCoder

2. Initialize the Analyzer

cc = ContentCoder(dicFilename='path/to/dictionary.dic', fileEncoding='utf-8-sig')

3. Analyze a Text Sample

text = "An abrupt sound startled him. Off to the right he heard it, and his ears, expert in such matters, could not be mistaken. Again he heard the sound, and again. Somewhere, off in the blackness, someone had fired a gun three times."
results = cc.Analyze(text, relativeFreq=True, dropPunct=True, retainCaptures=False, returnTokens=True, wildcardMem=True)
print(results)

Expected output:

{
  "WC": 23,
  "Dic": 5.4,
  "BigWords": 6.0,
  "Numbers": 3.0,
  "AllPunct": 0.0,
  "Period": 3.0,
  "Comma": 0.0,
  "QMark": 0.0,
  "Exclam": 0.0,
  "Apostro": 0.0
}

---

Main Functions & Usage

1. Analyze(text, **options)

Analyzes a given text and returns a dictionary of results.

Parameters:

• inputText (str): The text to analyze.
• relativeFreq (bool): If True, returns relative frequencies. Otherwise, raw frequencies.
• dropPunct (bool): If True, punctuation is removed before processing.
• retainCaptures (bool): If True, captures and stores wildcard-matched words.
• returnTokens (bool): If True, returns tokenized text.
• wildcardMem (bool): If True, speeds up wildcard processing by storing past matches.
• weightedMean (bool): If True, each category comes back as the mean of the weights of the terms that matched, rather than as a rate. See below.

Example Usage:

result = cc.Analyze("Hello world! This is a test sentence.", returnTokens=True)

Weighted dictionaries: rates vs. mean ratings

There are two different kinds of weighted dictionary out there, and they want two different answers.

When the weights are amounts — the eMFD, where a weight is roughly how much of a word belongs to a category — the default is what you want: the category score is how much of the text landed in it, with weights counted for as much as they say.

When the weights are ratings — concreteness norms, valence norms, age of acquisition — a rate is meaningless. What you want is the average rating of the words that had one. Set weightedMean=True and that's what you get:

result = cc.Analyze(text, weightedMean=True)

result['Concreteness']              # mean rating of the matched words, or None
result['_MatchCounts']['Concreteness']   # how many entries matched
result['_MatchedWC']['Concreteness']     # how many words those entries covered
result['_MatchedWC']['Concreteness'] / result['WC']   # share of text that was rated

Each dictionary entry counts once no matter how many words it spans, so a two-word entry rated 4.0 is a single observation of 4.0 rather than two. A category with nothing matched comes back as None rather than 0, because a text with no rated words in it doesn't have a concreteness of zero — it doesn't have one at all. relativeFreq is ignored in this mode.

Multi-word entries are consumed once

If your dictionary holds both ice cream and cream, the phrase is matched and then skipped past — the cream inside it is not counted a second time on its own. (Before 1.3.0 it was: the line meant to skip ahead was written as i += numberOfWords - 1 inside a for i in range(...), which does nothing in Python, so overlapping entries were double counted. It came over from the C# version, where it works.)

One thing to set when you load a norm set: pass keepZeroWeights=True to the constructor.

cc = ContentCoder(dicFilename='Lancaster Sensorimotor.csv', keepZeroWeights=True)

In a content-coding dictionary a weight of 0 means "this term is not in this category", so it gets dropped on load and that saves a lot of memory. In a set of norms it means the opposite — somebody rated the word and the rating was zero. A third of the words in the Lancaster sensorimotor norms have a gustatory strength of exactly 0, and dropping those would leave the mean taken over only the words that taste of something. The default stays False so nothing existing changes.

---

2. GetResultsHeader()

Returns a list of all available output categories.

Example Usage:

print(cc.GetResultsHeader())

Expected output:

["WC", "Dic", "BigWords", "Numbers", "AllPunct", "Period", "Comma", "QMark", "Exclam", "Apostro"]

---

3. GetResultsArray(resultsDICT, rounding=4)

Formats the results of Analyze() into a CSV-friendly list.

Example Usage:

text = "The government plays an important role."
result = cc.Analyze(text)
csv_row = cc.GetResultsArray(result)
print(csv_row)

Expected output:

[6, 4.3, 2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]

---

4. ExportCaptures(filename, fileEncoding='utf-8-sig', wildcardsOnly=False, fullset=True)

Exports wildcard-captured words and their frequencies to a CSV file.

Example Usage:

cc.ExportCaptures("captured_words.csv")

---

5. ExportDict2022Format(dicOutFilename, fileEncoding, **options)

Exports the loaded dictionary in LIWC-22 format.

Example Usage:

cc.dict.ExportDict2022Format("dictionary_2022.dicx")

---

6. UpdateCategories(dicTerm, newCategories)

Updates the categories associated with a dictionary term.

Example Usage:

cc.dict.UpdateCategories(dicTerm="happiness", newCategories={"positive_emotion": 1.0, "joy": 0.5})

---

Example: Processing a Large CSV File with tqdm

This script reads a large CSV file and processes each text in the "body" column.

import csv
from tqdm import tqdm
from contentcoder.ContentCoder import ContentCoder

cc = ContentCoder(dicFilename='dictionary.dic', fileEncoding='utf-8-sig')

with open("Comments.csv", "r", encoding="utf-8-sig") as csvfile, \
     open("Output.csv", "w", encoding="utf-8-sig", newline="") as csvfile_out:

    reader = csv.DictReader(csvfile)
    writer = csv.writer(csvfile_out)
    writer.writerow(["id"] + cc.GetResultsHeader())

    for row in tqdm(reader, desc="Processing", unit=" comments"):
        row_id = row["id"]
        text = row["comment_text"]
        result = cc.Analyze(text)
        csv_row = cc.GetResultsArray(result)
        writer.writerow([row_id] + csv_row)

print("Finished!")

---

License

MIT License © 2021