corpus-lens

Drop-in TF-IDF + rhetorical-fingerprint + embedding analysis for any research corpus, with a static-HTML dashboard.

You have a folder of papers, blog posts, working notes, lecture transcripts, or any other writing. You want to know:

• What is each paper actually about? (keywords + bigrams)
• What is its rhetorical posture? (hedging / assertion / empirical / theoretical / critical / quantitative density)
• Which papers are similar to which? (cosine on TF-IDF, top-K)
• Where does each paper sit in concept space? (UMAP / PaCMAP / PCA)
• How does it compare to public baselines? (arxiv categories, specific researchers)
• What clusters emerge? (force-directed similarity network)

corpus-lens gives you all of that from a single command, with a self-contained HTML dashboard you can open in any browser, host as a static site, or embed in your existing docs.

Zero hard dependencies for the core analysis (stdlib only). Embeddings need numpy + sklearn + optionally umap-learn / pacmap. No vector DB, no Docker, no LLM API calls, no telemetry.

Quickstart

pip install corpus-lens[all]

# Discover what's in your input directory + emit an editable manifest
corpus-lens prepare ./my-papers --output ./.lens

# Run the analysis (TF-IDF, fingerprint, similarity, keyword index)
corpus-lens analyze ./my-papers --output ./.lens

# Add 2D embeddings (UMAP / PaCMAP / PCA)
corpus-lens embed --output ./.lens

# Add external comparison baselines from arxiv
corpus-lens compare --output ./.lens \
    --against arxiv:cs.LG --against arxiv-author:Friston

# Build the dashboard and open it
corpus-lens dashboard --output ./.lens --title "My Lab" --serve

That's it. The dashboard at http://127.0.0.1:8765/dashboard.html will show:

• Per-paper view: top-20 TF-IDF keywords, distinctive vocab, bigrams, rhetorical fingerprint, top-5 most-similar papers
• Map: UMAP / PaCMAP / PCA scatter, coloured by programme / maturity / year
• Network: force-directed similarity graph
• Timeline: lane chart of papers over time, per programme
• Heatmap: programme × rhetorical-dimension grid
• Plots: configurable scatter, programme bars, length histogram
• Compare: side-by-side with arxiv baselines
• Search by term: full inverted index
• Leaderboard: sort papers by any metric

Input formats

corpus-lens accepts your corpus in three forms.

1. Directory of files (most common)

Drop .md, .html, .htm, .txt, or .pdf files into a directory. corpus-lens prepare will discover them, extract titles / dates / metadata, and emit a manifest you can edit. PDFs require pip install corpus-lens[pdf].

2. Frontmatter-annotated files

Add YAML-ish frontmatter to your markdown / HTML to override extracted metadata:

---
title: My Paper Title
date: 2026-06-18
programme: Statistical Mechanics
maturity: preprint
tags: koopman, perturbation, spectral theory
---

# My Paper Title

The rest of the content...

3. JSON manifest

For non-file-based corpora (database dumps, API exports), write a manifest directly:

{
  "items": [
    {
      "id": "paper-001",
      "title": "Whatever",
      "date": "2026-06-18",
      "programme": "Some Programme",
      "maturity": "preprint",
      "tags": ["tag1", "tag2"],
      "text": "The full plain-text body of the paper..."
    }
  ]
}

Then corpus-lens analyze --manifest ./manifest.json --output ./.lens.

See docs/INPUT_FORMATS.md for full schema details.

What the rhetorical fingerprint measures

Each paper gets a six-axis fingerprint in markers per 1000 tokens:

Axis	What it counts	Why it matters
hedging	"may", "might", "could", "perhaps", "suggests", ...	Epistemic caution, openness
assertion	"proves", "establishes", "demonstrates", "definitively", ...	Confidence, finality
empirical	"experiment", "dataset", "measure", "baseline", ...	Data-driven posture
theoretical	"theorem", "lemma", "proof", "axiom", ...	Formal-derivation posture
critical	"refutes", "fails", "mistaken", "naive", ...	Argumentative / refutational tone
quantitative	numbers with units (10ms, 95%, 3.2kHz)	Numeric specificity

The lexicons are tunable; pass --lexicons to override or extend via the Python API. See docs/INTERPRETATION.md for guidance on reading the fingerprint.

Python API

The CLI is a thin wrapper. The full API is small and importable:

from corpuslens import extract, analyze, embed, compare, dashboard

items = extract.from_directory("./my-papers")
result = analyze.run(items)
embeddings = embed.run(result, methods=["pca", "umap", "pacmap"])
baselines = compare.run(["arxiv:cs.LG", "arxiv-author:Friston"], our_analysis=result)
dashboard.build("./.lens", title="My Lab")

Each function takes plain dicts/lists and returns plain dicts/lists. No frameworks, no inheritance, no DI.

Preparation guide

To get the most out of corpus-lens, spend ten minutes on preparation. See docs/PREPARATION.md for the full guide. The short version:

1. One file per paper. Don't pre-concatenate. The unit of analysis is the paper.
2. Use frontmatter. Title and date in YAML at the top of every file, even if the extractor would auto-detect them.
3. Pick a programme label. A 1-3 word category for each paper. Loose groupings are fine; this is just colour-coding in the dashboard.
4. Strip boilerplate. Page headers, footers, navigation, references sections, and author bios skew the keyword analysis. The HTML extractor drops these automatically; for markdown, put boilerplate in fenced blocks (\```) so it's excluded.
5. Decide on maturity tags. draft, preprint, published, working are useful axes for the dashboard's colour-by-maturity view.

Comparison baselines

corpus-lens compare pulls abstracts from arxiv and applies the same fingerprint so you can position your corpus against published baselines.

corpus-lens compare --against arxiv:cs.LG          # ML
corpus-lens compare --against arxiv:math.PR        # probability
corpus-lens compare --against arxiv:q-bio.NC       # neuroscience
corpus-lens compare --against arxiv:cond-mat.stat-mech  # stat mech
corpus-lens compare --against arxiv-author:Friston
corpus-lens compare --against arxiv-author:Bengio
corpus-lens compare --against arxiv-query:cat:math.DS+AND+abs:Koopman

Each baseline becomes a row in the dashboard's Compare tab. Hover for the per-1k-token density of each rhetorical axis.

Output files

After analyze + embed + compare, your .lens/ directory contains:

.lens/
  manifest.json             # editable input description
  corpus_analysis.json      # per-paper meta + keywords + fingerprint
  corpus_keyword_index.json # inverted index: term -> [{paper, tf}]
  corpus_similarity.json    # paper -> top-K most-similar
  corpus_embedding.json     # 2D coordinates per paper, per method
  comparison_corpora.json   # external baselines
  dashboard.html            # the dashboard (loads the JSON siblings)

All JSON. All plain. Open in any browser, host as a static site (GitHub Pages, Netlify), or pipe into another tool.

Hosting the dashboard

The dashboard is a single HTML file plus its JSON siblings. Drop the whole .lens/ directory anywhere that serves static files:

# GitHub Pages
gh repo create my-corpus --public
cp -r .lens/* docs/
gh-pages -d docs

# Netlify drop: drag .lens/ onto netlify.app/drop
# Vercel: vercel deploy ./.lens
# nginx / Apache: copy .lens/ to your webroot

No build step. No server-side code.

License

MIT.