docfriction

Friction logs for documentation pages, generated by walking a page section by section and asking TypeSafe Jev a fixed set of typed questions about each one.

A friction log is the write-up a developer advocate produces after going through a quickstart as a new user: one entry per step, what they expected, where they got stuck, how bad it was. Teams get a lot out of them and rarely have time to write them, so most pages never get one and the ones that exist go stale on the next edit.

docfriction produces the same document from a URL or a Markdown file:

• One step per heading — the page is split at headings; each step carries its heading path, prose, code blocks and links, plus a summary of the previous section as context
• One Jev call per step — friction type (a choice over eight categories plus no_friction), severity (a score over four reader outcomes), and yes/no checks for prerequisites, expected result, undefined terms, code/prose match and unexplained placeholders, batched into a single request
• Deterministic checks for what Jev can't do — dead links (--check-links), code blocks with no language tag, stub sections, placeholder detection
• Calibrated output — every finding carries Jev's probability and confidence; answers in the uncertain band are labelled "needs review" instead of being reported as findings
• CI gate — --fail-on-severity exits 2 when any step scores past your threshold; a ready-made GitHub Actions workflow is in examples/
• Cheap enough to run on every change — Jev charges $0.042 per million input tokens and answers in roughly 100 ms, so a 40-section page costs well under a cent and finishes in a few seconds

$ docfriction https://docs.example.com/quickstart --check-links

# Friction log: Quickstart
Steps: 9 · steps with friction: 2 · max severity: 2.4/3 · Jev tokens: 4,120 in, 210 out (about $0.0002; output is free)

| # | Step                         | Sentiment     | Severity | Friction                                  |
|---|------------------------------|---------------|----------|-------------------------------------------|
| 1 | Quickstart                   | 😀 smooth     | 0.2      | –                                         |
| 2 | Quickstart > Install         | 😀 smooth     | 0.3      | –                                         |
| 3 | Quickstart > Configure       | 🛑 blocked    | 2.4      | missing_prerequisite (0.81), dead_link    |
| 4 | Quickstart > First request   | 😐 pause      | 1.1      | –                                         |
...

A full report is in examples/sample-friction-log.md. The notes on Jev that shaped the design are in docs/research.md.

Installation

Not on PyPI yet. Install from GitHub:

uv tool install git+https://github.com/Cyvid7-Darus10/docfriction   # or: pipx install git+https://github.com/Cyvid7-Darus10/docfriction
export TYPESAFE_API_KEY=...                                           # from https://typesafe.ai

Requires Python 3.10+. --dry-run works without a key. To work on it:

git clone https://github.com/Cyvid7-Darus10/docfriction
cd docfriction && uv sync
uv run docfriction --help

Usage

# A live docs page, Markdown report to stdout
docfriction https://docs.example.com/quickstart

# A local file, JSON report to disk, and check every external link
docfriction docs/quickstart.md --format json --out friction.json --check-links

# See how the page splits into steps before spending any tokens
docfriction https://docs.example.com/quickstart --dry-run

# CI: exit 2 if any step scores 2.5 or higher on the 0-3 severity scale
docfriction docs/quickstart.md --fail-on-severity 2.5

Exit codes: 0 ok, 1 fetch or API error, 2 severity threshold exceeded.

Why Jev

Jev doesn't generate text. You send it a state and a set of typed questions (yes/no, pick one, rate on a scale) and it returns one typed answer per question with a probability distribution and a confidence number. The judgments in a friction log are all of that shape:

Because the answers are numbers, thresholds decide what counts as a finding and CI can fail on severity. Jev can't count, compare versions, follow links or explain itself, so those parts are ordinary code in checks.py.

How it works

1. Fetch. URLs are requested with an Accept header that prefers Markdown, which many docs sites serve. HTML is reduced to its <article>/<main> content and converted to Markdown with fenced, language-tagged code blocks.
2. Segment. The Markdown is split at ATX (#) and setext (underlined) headings. Each segment keeps its heading path, prose, fenced and indented code blocks, and links. Front matter and HTML comments are dropped. Prose over 6,000 characters is truncated so the Jev state stays small; Jev degrades on large irrelevant context.
3. Build state. Each step is evaluated the way a reader meets it: a summary of the previous section, the section text, the code blocks, and any placeholders the regex found.
4. Ask Jev once per step. Every rubric question goes in one request. Questions that only apply to actionable steps are still asked, and only applied when is_actionable comes back likely.
5. Interpret. Probabilities become findings through Thresholds in evaluate.py. A yes/no check below 0.4 is a finding; between 0.4 and 0.6 it is "needs review". A friction type is reported when its probability is at least 0.5 and marked for review when Jev's confidence is under 0.4.
6. Report. Markdown laid out like a hand-written friction log (summary table, then a walkthrough with a sentiment per step), or JSON for tooling.

Sentiment maps the 0-3 severity score to the marker a human would put in the margin: 😀 smooth (under 0.75), 😐 pause (under 1.5), 😠 frustrated (under 2.25), 🛑 blocked.

Library use

from docfriction import EvaluateOptions, JevClient, evaluate_document, load_document, render_markdown

document = load_document("https://docs.example.com/quickstart")
with JevClient() as client:
    log = evaluate_document(document, client, EvaluateOptions(check_links=True))
print(render_markdown(log))
for step in log.friction_steps:
    print(step.segment.title, step.severity, [f.check for f in step.confirmed_findings])

Tuning the rubric

Everything Jev is asked lives in rubric.py. The wording follows TypeSafe's guidance: judgments in instructions, answers in criteria, score levels that describe situations rather than degrees, a no_friction option so the choice is never forced, and backticked references to state fields. If your docs have a house rule (say, every step must end with expected output), add a Noul for it and a line in NOUL_FAILURE_DETAIL.

The default thresholds are starting points. Run docfriction on a few pages you know well, compare its findings with your own, and adjust Thresholds before wiring it into CI.

Status

• Not yet validated against human friction logs. Precision and recall per rubric item are unknown.
• Jev returns probabilities, not reasons. The report says which rubric item fired and how likely; a writer still has to read the section to decide what to change.
• Jev reads literally and isn't hardened against prompt injection, so a page can influence its own score. This is a quality tool, not a security control.
• One heading is one step. A tutorial under a single heading gets one coarse evaluation; a page of tiny headings gets many stub findings. --dry-run shows the split before you spend tokens.

Development

uv sync --all-groups
uv run pytest --cov=docfriction     # coverage gate is 80% in CI
uv run ruff check src tests && uv run ruff format src tests
uv run mypy src                     # strict

The Jev client is a thin httpx wrapper over the documented POST /v1/systemone endpoint with backoff on 429/529. Answers are validated against the questions that were asked, so a choice Jev was never offered can't reach a report. All tests run against httpx.MockTransport; no network or key is needed.

Contributing

Bug reports, rubric ideas and pull requests are welcome. CONTRIBUTING.md covers setup, the checks CI runs, and the rules for changing what we ask Jev. Security issues go through SECURITY.md. Releases are listed in CHANGELOG.md.

License

MIT. Authored by Cyrus David Pastelero.