tfidf-zones

CLI tool that classifies terms in text documents into three zones based on TF-IDF and document frequency:

• Too Common — high document frequency (df > 0.2N)
• Goldilocks — high TF-IDF score within a moderate DF band (3 ≤ df ≤ 0.2N, tfidf ≥ Q95)
• Too Rare — low document frequency (df < 3)

Useful for stylometric analysis, authorship attribution, and understanding term importance.

Install

poetry install

Usage

# Analyze a single file
poetry run tfidf-zones --file novel.txt --output results.csv

# Analyze with bigrams
poetry run tfidf-zones --file novel.txt --ngram 2 --output results.csv

# Analyze a directory of .txt files
poetry run tfidf-zones --dir ./texts/ --output results.csv

# Show top 25 terms per zone with custom chunk size
poetry run tfidf-zones --file novel.txt --top-k 25 --chunk-size 500 --output results.csv

Recipes

Find content-word bigrams across a corpus:

poetry run tfidf-zones \
  --dir ./texts/ --limit 100 --output results.csv \
  --no-chunk --wordnet --ngram 2 --no-ngram-stopwords

Combines --wordnet (only real English words), --ngram 2 (bigrams), and --no-ngram-stopwords (discard bigrams containing stop/function words like "of_the") to surface meaningful two-word terms.

Find content phrases (trigrams and above):

poetry run tfidf-zones \
  --dir ./texts/ --output results.csv \
  --no-chunk --wordnet --ngram 3 --no-ngram-stopwords

Increase --ngram to 3, 4, or 5 to find longer phrases. The stopword filter removes any n-gram where at least one token is a stop word or function word, so only content-rich phrases survive.

Corpus analysis with post-processing filters:

poetry run tfidf-zones \
  --dir ./texts/ --output results.csv \
  --no-chunk --wordnet --min-df 2 --min-tf 2

Use --min-df and --min-tf to remove terms that appear in too few documents or have too few total occurrences, reducing noise from hapax legomena.

Options

Either --file or --dir is required (not both).

How It Works

Text is tokenized, split into chunks, and scored with TF-IDF. Chunking a single document into sub-documents prevents IDF from collapsing to a constant. Terms are then bucketed into zones by their document-frequency percentile.

Engines

The default engine is a pure-Python implementation that requires no heavy dependencies and generally produces better results for zone analysis. It uses smooth IDF (log((1+N)/(1+DF)) + 1) with full control over tokenization, n-gram generation, and scoring.

A scikit-learn engine (--scikit) is also available for users familiar with the scikit-learn ecosystem or who want to compare results against its TfidfVectorizer. Both engines use the same IDF formula and produce comparable output. scikit-learn is an optional dependency — install it with:

pip install tfidf-zones[scikit]

or if using Poetry:

poetry install -E scikit