CodeAudit

A classical NLP pipeline that scans Python codebases, extracts comment blocks, and classifies each one by quality — with no LLMs, no API calls, and no GPU required.

---

What It Does

You point it at any Python project. It reads every file, finds all comments, groups them into logical blocks, classifies each block, and generates a ranked report.

Each comment block gets one of four labels:

Label	Meaning
informative	Explains why or what — adds real value
low_quality	Vague, redundant, or restates the code
directive	TODO, FIXME, HACK, NOTE, WARNING
outdated	References removed APIs, old versions, legacy code

---

Why Classical NLP and Not an LLM

	This tool	LLM-based approach
Inference per block	< 2ms	2000–10000ms
50K file audit	~ 50 seconds	~ 14 hours
Cost	$0	$5–$50 per run
Works offline	✅	❌
Reproducible	✅ 100%	❌ varies
Install size	~ 50MB	2GB+

For a measurement problem across thousands of files, classical NLP is the correct architecture — not a compromise.

---

Installation

pip install codeaudit
python -m spacy download en_core_web_sm

---

Usage

codeaudit scan ./my_project --output ./results

Open results/codeaudit_report.html in your browser for the full visual report.

---

Output

Three files are generated in your output folder:

results/
├── codeaudit_report.html     ← visual dashboard, open in browser
├── block_report.csv          ← every comment block with label + confidence
└── file_report.csv           ← per-file quality summary

block_report.csv

file_path        | block_id | start_line | end_line | text                        | label       | confidence
flask/app.py     | 1        | 12         | 14       | Retries up to 3 times...    | informative | 0.91
flask/app.py     | 2        | 23         | 23       | TODO: fix auth flow         | directive   | 0.96
flask/helpers.py | 1        | 8          | 9        | this does the thing         | low_quality | 0.88

file_report.csv

file_path        | total_blocks | informative_pct | low_quality_pct | directive_pct | outdated_pct | dominant_label
flask/app.py     | 42           | 61.9            | 21.4            | 14.3          | 2.4          | informative
flask/helpers.py | 18           | 33.3            | 55.6            | 11.1          | 0.0          | low_quality

---

Model Performance

Trained on 800 manually labeled comment blocks from 5 major open-source Python repositories — Flask, Requests, Django, pytest, click.

Inter-rater reliability validated at Cohen's Kappa = 0.74 before training. Evaluated on 160 held-out blocks the model never saw during training.

Model	Accuracy	F1 (weighted)
Logistic Regression	0.956	0.957
Linear SVM	0.950	0.949
Naive Bayes	0.825	0.813

Default model is Logistic Regression.

A note on class distribution

The dataset reflects real-world comment distributions:

low_quality    69%
informative    19%
directive      10%
outdated        1%

Overall F1 is influenced by this imbalance. Per-class F1 from the classification report is the more meaningful metric — particularly for minority classes like outdated. All models were trained with class_weight=balanced to compensate.

---

NLP Pipeline

raw comment text
      ↓
camelCase / snake_case splitting
      ↓
lowercase + noise removal
      ↓
tokenization
      ↓
stopword removal  (NLTK)
      ↓
lemmatization     (NLTK WordNet)
      ↓
TF-IDF vectorization  (ngram 1–3, tuned vocabulary size)
      ↓
hand-engineered features  (readability, keyword flags, structural)
      ↓
Logistic Regression classifier
      ↓
label + confidence score

---

Feature Engineering

Two types of features are combined into one matrix:

TF-IDF features

• Unigrams, bigrams, trigrams on cleaned comment text
• Sublinear TF normalization
• Vocabulary size tuned via grid search

Hand-engineered features

• Flesch-Kincaid readability score
• Gunning Fog index
• Word count
• Keyword flags — todo, fixme, hack, note, warning, deprecated, bug
• Block line count
• Code token ratio — how much code leaked into the comment
• Average token length

---

Dataset

No existing labeled dataset existed for code comment quality classification. It was built entirely from scratch:

• Cloned 5 major open-source Python repositories
• Extracted comment blocks using Python's tokenizer module
• Manually labeled 800 blocks across 4 categories
• Validated inter-rater reliability with a second annotator
• Cohen's Kappa = 0.74 confirmed before training began
• Only blocks where both annotators agreed were kept

Repositories used for training: Flask, Requests, Django, pytest, click

---

Validation

• 80/20 stratified train/test split
• Evaluated on 160 held-out blocks never seen during training
• 5-fold cross-validation during hyperparameter tuning
• Calibration curves confirm confidence scores are meaningful
• Tested on held-out repositories not used in training or labeling

---

Extending This Tool

Add more languages Swap the Python tokenizer for tree-sitter to support Java, Go, Rust.

CI/CD integration Run on every pull request. Flag files where low_quality percentage rises above a configured threshold.

Drift detection Run on the same repo at two points in time. Track whether comment quality improves or degrades across releases.

---

Requirements

Python     >= 3.9
scikit-learn >= 1.3
pandas       >= 2.0
spacy        >= 3.7
nltk         >= 3.8
textstat     >= 0.7
scipy        >= 1.11
matplotlib   >= 3.7
jinja2       >= 3.1
click        >= 8.1

---

License

MIT License.