aidetect 0.3.0

 █████╗ ██╗      ██████╗ ███████╗████████╗███████╗ ██████╗████████╗
██╔══██╗██║      ██╔══██╗██╔════╝╚══██╔══╝██╔════╝██╔════╝╚══██╔══╝
███████║██║█████╗██║  ██║█████╗     ██║   █████╗  ██║        ██║
██╔══██║██║╚════╝██║  ██║██╔══╝     ██║   ██╔══╝  ██║        ██║
██║  ██║██║      ██████╔╝███████╗   ██║   ███████╗╚██████╗   ██║
╚═╝  ╚═╝╚═╝      ╚═════╝ ╚══════╝   ╚═╝   ╚══════╝ ╚═════╝   ╚═╝

🔍 What is this

A command-line tool that reads a .docx or .txt and scores each paragraph 0–1 on how AI-generated it reads, using the desklib/ai-text-detector-v1.01 DeBERTa model — the one sitting at #1 on the RAID benchmark. Everything runs on your own machine; after the first model download it never touches the network. You point it at your Extended Essay, it tells you which paragraphs sound like a language model wrote them.

The point isn't to cheat a detector. It's the opposite: I write my own drafts, and sometimes my own honest prose still trips these classifiers because that's what earnest formal writing looks like to them. This flags those paragraphs so I can reword before a teacher runs Turnitin and has an awkward conversation with me about it.

It is directional, not oracular. A high score means "reword this," not "you're caught." It is not, and cannot be, the number Turnitin shows a teacher.

nick@aidetect:~$ aidetect score EE-clean.txt
P  1   0.08  [##------------------]
P  2   0.71  [##############------]  <-- AI-ish
average AI score: 0.34   |   1/6 paragraphs flagged
reminder: directional only, not a Turnitin score.

🧠 The detection engine

🚀 Run it

uv tool install aidetect      # or: pipx install aidetect

aidetect                                   # list the seven subcommands
aidetect count "draft.docx" --limit 4000   # IB word count, by section
aidetect count "draft.docx" --json         # same, as one JSON object
aidetect extract "draft.docx"              # -> "draft prose.txt", what got counted
aidetect score "draft.docx"                # score a whole draft, paragraph by paragraph
aidetect score "draft.docx" --segments     # Turnitin-shaped: % of prose in flagged windows
aidetect score --text "one sentence"       # score a single string
aidetect bino  "draft.docx" --mlx --pair gemma
aidetect check "draft.docx"                # both detectors, worst opinion wins

count and extract are instant and need no model. score and bino need a machine that can hold a transformer: built and tested on an 18GB Apple Silicon Mac, MPS-accelerated. Their first run downloads the model and will sit there for a minute — that's normal, not a hang. Every run after is fast and offline.

On Apple Silicon the Gemma 4 MLX pair installs automatically. Elsewhere it is skipped and the Qwen pairs still work.

--json

count takes --json and prints exactly one object on stdout, nothing else:

{"sections": [{"title": "Introduction", "words": 812, "level": 1},
              {"title": "Analysis", "words": 0, "level": 1},
              {"title": "Porter's Five Forces", "words": 1021, "level": 2}],
 "total": 3940, "limit": 4000, "over": -60}

Every key is always present. limit and over are null when no --limit was given — that means "does not apply", not "could not be read". A draft with no prose is an empty sections list and exit 0. Errors go to stderr with a non-zero exit, so consumers branch on the exit code rather than parsing error text.

words is a section's own words, never its children's, so sum(words) equals total and nothing double-counts. The table you see rolls sub-sections up into their parent for display; do the same yourself with level if you want subtotals.

What counts

Excluded by position, not by wording: everything before the first heading (the cover page), the Table of Contents section, headings themselves, Figure 3: ... captions, tables, footnotes, and everything from the Bibliography heading on. In-text citations are stripped from the paragraphs that survive.

Block quotes and body bullet lists do count — they are assessed prose. A caption needs its colon to be dropped, so Figure 4 shows revenue rising is counted while Figure 4: Revenue, 2021–2025 is not. A draft with no headings at all is counted whole, with a note, rather than reporting a confident zero.

🔩 Under the hood

flowchart LR
    A[".docx"] --> B["walk()<br/>structure: drop cover, contents,<br/>headings, captions, bibliography"]
    B --> C["aidetect count<br/>strip citations,<br/>total by section"]
    B --> P["is_prose()<br/>style floor: >= 25 words,<br/>no bullets or scaffolding"]
    B --> S["sentence windows<br/>7 sentences, stride 3<br/>no length floor"]
    P --> D1["desklib DeBERTa<br/>per paragraph"]
    P --> BN["Binoculars<br/>Gemma 4 pair<br/>per paragraph"]
    S --> D2["desklib DeBERTa<br/>per window"]
    D1 --> O1["aidetect score<br/>0-1 + flags"]
    D2 --> O2["score --segments<br/>% of prose flagged"]
    BN --> O3["aidetect bino<br/>vs calibrated threshold"]
    O2 --> K["aidetect check<br/>worst opinion<br/>per sentence"]
    O3 --> K

One walker decides what is in the document; everything downstream is a different question asked of the same paragraphs. A 6-word sentence is a word the examiner counts and noise to a paragraph scorer, which is why count and is_prose are separate filters rather than one.

The two scoring paths differ on purpose. Paragraph mode applies the 25-word floor, because a fragment scores as noise on its own. Segment mode drops it and slides overlapping windows instead, so those same short connective sentences get scored inside a window with their neighbours — and formulaic linking prose is exactly what detectors flag. check runs the segment path and Binoculars together and takes the worse verdict per sentence.

🔭 Binoculars: shelved, then revived by Gemma 4

Binoculars is a training-free detector: run text through two LMs that share a tokenizer (a base "observer" and an instruct "performer") and divide perplexity by cross-perplexity. Its designed model pair is Falcon-7B ×2 (~28GB) — too big for an 18GB Mac, so the fallback was a small same-family pair (Qwen2.5-0.5B or 1.5B) that fits.

aidetect calibrate scores a labelled set — mine is 12 real pre-2020 IB Extended Essay paragraphs vs 12 LLM-written ones on the same topics — and finds the best separating threshold. Measured across pairs:

The Qwen pairs sit near a coin flip: their human and AI score clusters almost completely overlap, because the perplexity gap Binoculars exploits is sharp in larger models and mush in sub-2B ones. That was the original negative result. Swapping in a Gemma 4 pair opens a clean gap (human mean 0.90 vs AI 0.69) and separates the set at 92%. Gemma 4 ships as a multimodal checkpoint, so --mlx quantizes it to 4-bit and runs it text-only through mlx-vlm, fitting the 18GB Mac in ~6GB:

aidetect bino IA-clean.txt --mlx --pair gemma   # uses the shipped threshold

# refit the threshold on your own labelled set
aidetect calibrate --human-dir corpora/human --ai-dir corpora/ai --mlx --pair gemma

The AI class has to be a fair adversary

aidetect generate builds the AI half of a calibration set by calling a hosted model through NVIDIA NIM with no system prompt and no style guidance — only a topic and a word count:

export NVIDIA_API_KEY=nvapi-...     # you set it; aidetect only reads the env var
aidetect generate --topics corpora/human-tech/manifest.json \
                  --out-dir corpora/ai-tech --prefix ta --seed 7

By default it reads NIM's live catalog and samples popular models across vendors, one per topic — twelve topics typically means twelve different models from twelve vendors, so the class carries a spread of tokenizers, architectures and training mixes rather than one model's habits. Models that NIM has stopped serving are dropped rather than 404-ing mid-run, --seed makes an assignment reproducible, and --model pins exact ids instead. The Gemma family is never sampled: it is the detector's own pair, and scoring Gemma output with a Gemma observer/performer would flatter the detector.

This exists because of a real failure. An earlier AI set was written by an assistant carrying house style rules (write tightly, no em dashes, active voice) and came out stylometrically indistinguishable from the human class it was supposed to oppose — mean sentence length 25.8 against the humans' 24.5, where a properly untuned AI class ran to 35.1. A too-human AI class pulls the clusters together, lowers the threshold and makes the detector more lenient, which is the false-negative direction this tool exists to avoid. That set is quarantined in corpora/ai-tech/ with the numbers written down.

Repeat --model to rotate models across topics so the class is not one model's quirks, and prefer a family different from the detector's own pair. The output manifest records the model, temperature and exact prompt template for every sample, so any threshold fitted on it can be audited.

Genre thresholds: --tag

The boundary is not only per model pair, it is per genre. Fitted on 12 real pre-2013 technical Extended Essay paragraphs (maths, CS, physics, chemistry, ITGS), human technical prose means 0.828 where humanities prose means 0.901. The humanities threshold of 0.826 therefore sits almost exactly at the mean of genuine human technical writing, so a maths or CS essay flags roughly half its paragraphs no matter who wrote it.

--tag keeps a second calibration beside the first instead of overwriting it:

aidetect calibrate --human-dir corpora/human-tech --ai-dir corpora/ai-tech \
                   --mlx --pair gemma --tag tech      # -> threshold-gemma-mlx-tech.json
aidetect check "IA Graph Theory.docx" --tag tech      # judged against technical prose

Measured on four of my own drafts, moving to the technical threshold cut the flag rate on the maths IA from 28% to 13% and the CS IA from 51% to 32%, while the two humanities drafts (3% and 13%) did not move at all.

Both separate 92%, so the genre effect is a shift in level, not a loss of discrimination: technical prose scores lower for everyone, and once the threshold moves with it the detector works just as well. An earlier run measured 79% here and was wrongly read as "Binoculars is weak on technical prose" — that number came from a contaminated AI class, and the story is in corpora/ai-tech/README.md.

desklib stays the primary detector; Binoculars is now a usable second opinion rather than a dead end. The calibration sets are not shipped with the package — clone the repo to reproduce the numbers, or point --human-dir/--ai-dir at your own. Your fitted threshold lands in ~/.config/aidetect and takes precedence over the shipped one, so it survives an upgrade.

🧪 The peer set: genre context, not a human class

The 12 human samples are Extended Essays: English, History, Biology, Physics, Philosophy. A CS IA is a different animal, all database schemas, GUI components and method-by-method justification, and technical prose is inherently more predictable token-by-token, which drags perplexity-ratio scores down no matter who typed it. So a CS IA scoring below the EE human mean means less than it looks.

corpora/peer/ holds 12 paragraphs of real IB Computer Science IA prose (5 projects, 5 authors: sudokuMaster, IBOrganizer, MyCalendar, and two IBO-published new-syllabus specimens). Measured against the same anchors:

The genre gap is real and it is about 0.07 on Binoculars, measured against human-tech — twelve technical EE paragraphs that are provably pre-AI, which is the anchor peer could only approximate. peer sitting at 0.85, between the two verified human classes, is consistent with that gap rather than evidence of anything about the students who wrote it.

Score a technical draft with --tag tech, which encodes exactly this shift.

It is not a human class and it never fits a threshold. Every source postdates ChatGPT; three of the five were written in 2025. None carries an authorship attestation, and in 2025 a fair share of student IAs were not written unaided. Fold that into human/ and any AI-assisted sample drags the mean down, lowers the threshold, and the tool starts clearing drafts for the wrong reason, a detector that reassures instead of measures. aidetect calibrate reads only the two folders you name, so peer/ stays out of threshold fitting by construction, not by discipline.

What it can tell you: "my prose scores like other IAs in this genre." What it can never tell you: "my prose is human." Matching a set you cannot vouch for proves you are not an outlier, nothing more.

Stack: python · pytorch · transformers · mlx-vlm · desklib DeBERTa

The cross-check that matters is built in: aidetect check runs both detectors and takes the worse verdict. Ejhfast/fast-ai-detector is a separate, lighter tool in its own repo, not vendored and not part of check's verdict — reach for it only when you want a read from a model neither detector shares.