indexgap

Lint your programmatic SEO pipeline — from keywords to indexed pages.

Русская версия

You generated three thousand pages from a dataset. They're on disk, they're in the sitemap, and there's no traffic. indexgap tells you where they were lost.

Pure Python 3.9+ standard library. No dependencies, no API keys, no paid subscriptions. Nothing leaves your machine without an explicit flag.

Why this exists

Plenty of tools audit a page. A generated pipeline breaks differently — systematically and quietly:

• a page is in the sitemap, but no internal link points to it, so the crawler never arrives;
• three hundred pages differ by five words, and the search engine keeps one;
• the model wrote "12 years in business" and "3,500 orders" — neither number exists in any row of your data, but it reads convincingly;
• lastmod equals the build date on every page, so it means nothing;
• two keywords with the same intent produced two competing pages.

None of this is visible page by page. Each one looks fine on its own.

Install

pipx install indexgap

pipx keeps the tool in its own environment and still puts indexgap on your PATH — pip install into a system Python is refused outright on most current distributions. If you do not have it: python3 -m pip install --user pipx. To work from the main branch instead: git clone the repository and pip install -e .

Then, once per project:

cd ~/projects/my-site
indexgap init

init reads the project and records what makes it different: where the pages are, the site URL, the content type, the dataset. It installs skills into .claude/skills/ so your coding agent picks them up on its own, and adds the working files to .gitignore.

Nothing project-specific is ever copied between projects — it is detected fresh each time. The IndexNow key in particular is never carried over: it is bound to one domain by a file at that site's root, and a borrowed key returns 403. indexgap init --key mints a new one for this project.

After that, the daily command is just:

indexgap check

Commands

indexgap init                       # install into this project
indexgap plan keywords.csv          # audit the keyword set before generating
indexgap check                      # everything local: text, structure, machine-readability
indexgap brief --write              # turn the findings into work orders beside the pages
indexgap sitemap --out-dir ./public # sitemap with sharding and an honest lastmod
indexgap notify --key <your-key>    # tell IndexNow what actually changed
indexgap doctor --sitemap ./public/sitemap.xml --indexed gsc.csv
indexgap portfolio projects.json    # every site you own, in one run
indexgap profiles                   # what the content-type presets change
indexgap cite --domain example.com  # do AI answers cite you? (your own API keys)

Output is English by default, Russian with --lang ru (or INDEXGAP_LANG, or your system locale). That includes the HTML report, every finding description, --help, and the skills indexgap init installs into your project.

Understands both built HTML and Markdown sources with frontmatter. Input file encoding is detected, not assumed: UTF-16, BOM, and the cp1251 CSVs that Russian Excel produces all read correctly. Exports are read as they come — CSV with any delimiter, XLSX straight from Ahrefs or Semrush without re-saving, JSON, NDJSON, an XML sitemap, or a plain list of URLs — and the keyword column is found whether the export calls it Keyword, Фраза, Запрос or Search Term.

What it checks

Before generating — exact duplicate keys, slug collisions and same-intent keys. That last one matters most: two keywords meaning the same thing will produce two competing pages, and it is cheaper never to create the second.

After generating — the text

After generating — the structure

Near-duplicates (exact pairwise below 400 pages, MinHash + LSH above), share of unique text measured by bigrams rather than words, thin pages, orphans, click depth, noindex, nosnippet, foreign canonicals, missing or duplicate H1, duplicate titles and descriptions.

Machine readability for AI search

Snippet controls (nosnippet, max-snippet:0), robots.txt rules for OAI-SearchBot, PerplexityBot, ClaudeBot, GPTBot and Google-Extended (they are not interchangeable — blocking OAI-SearchBot removes you from ChatGPT search answers, while Google-Extended does not affect AI Overviews), empty JS shells, a direct answer in the first paragraph, question-shaped subheadings, valid JSON-LD that matches the visible text, machine-readable dates and author.

At publish time — sitemap sharding past 45,000 URLs and a lastmod that changes only when the text, title or description changed. Editing one menu item does not mark the whole site as modified. IndexNow sends only what changed.

From findings to work orders

A report answers "what is wrong with me". indexgap brief answers "what do I do" — it lays the same findings out as markdown files next to the pages they belong to, each with an imperative fix, the thresholds the fix has to meet, and the dataset row as the only permitted source of numbers.

indexgap brief --dataset keywords.csv        # dry run: says what it would write
indexgap brief --dataset keywords.csv --write

On the live 2,970-page catalogue the check reports 6,074 findings and brief writes 966 work orders. The difference is three placement rules, and they are the whole point:

• A property of the template gets one brief, not 2,919. no-question-headings fired on 2,919 of 2,970 pages. That is one edit to one template, and 2,919 identical tasks would bury the 51 pages that actually differ. The same holds per language: a finding on every page of the Chinese version is 10% of the site but still one template fix.
• Near-duplicates are fixed as a group. 588 duplicate findings are 72 groups. One page out of a group of 17 cannot be fixed alone — the brief goes to the group: keep one, pull the rest apart by intent, and never link them to each other.
• robots.txt, markup and hreflang clusters belong to the site, not to a page, and live in their own file.

--limit (50 by default) writes the heaviest pages first; 892 briefs is not a task list, it is a second report. Nothing is created without --write.

The package writes no text, on purpose. Its central check compares the numbers on a page against the dataset row that produced it. If the package supplied those numbers itself, the check would be checking its own output and would always be green. So brief states the task and stops; a person or an agent writes the words.

The funnel

The reason to install it at all:

Generated                 58
Indexable                 56   (−2: noindex or a canonical pointing elsewhere)
In sitemap                56
In at least one index     28   (−28: the engine knows the URL and didn't add it)

Why pages are not indexed:
    7  orphaned or unreachable  → link them from hub pages
   12  deeper than the click budget → move them up
    9  no local explanation → check status in Search Console

Index data comes from ordinary exports — no API keys. A webmaster panel is the direct source (Search Console, Bing Webmaster Tools, Yandex.Webmaster, Naver, Seznam), but not everyone has one, so exports from Ahrefs, Semrush, Serpstat, Moz, Screaming Frog, Sitebulb, JetOctopus, OnCrawl, Netpeak, GA4, Matomo, Plausible — or a plain list of URLs — are read too, in CSV, XLSX, JSON, NDJSON or XML.

They are not interchangeable, and the tool refuses to pretend otherwise. A panel answers "does the engine know this page". Analytics proves a page is indexed, but only for pages someone actually visited — its silence proves nothing. A crawler proves reachability, not indexation. Ahrefs and Semrush are their index, not Google's. So the funnel step is renamed to match the evidence — "at least in one index" versus "known to a third-party service" — and when a crawler export sits next to a panel, the report says out loud that the step count is higher than real indexation. Engine-vs-engine comparison runs over panels only.

Ask for all the panels you have, not just Google. A page missing everywhere is a technical problem. A page missing only in one engine was crawled and accepted by the others, which makes it a quality or speed question that technical fixes rarely solve. Without the split the two look identical and people fix the wrong thing.

AI citations are a step of their own. Bing Webmaster Tools → AI Performance exports the pages Microsoft Copilot cited and how often. Pass it like any other export (--indexed ai-performance-pages.csv; the Citations column is enough for the tool to recognise it) and the funnel gains a last step — cited — after the index. It is deliberately not merged into the index step: the export is a sample, 93 pages on a site with over a thousand indexed, and merged in it would declare everything else unindexed. What it adds that nothing else can: pages the AI still cites although you closed them from search, and cited pages you left out of the sitemap. The queries export from the same screen holds no page addresses; the tool says so and tells you which tab to export instead.

Every sitemap, and everything the engine knows that you did not generate. --sitemap can be repeated, and when robots.txt declares sitemap files you did not pass, the tool names them — on a live site one sitemap.xml held 13 URLs and the three declared files together held 94. Addresses the engine reports but the site does not contain are listed by meaning: other hosts (staging, admin, internal tools — the ones to close), pages that no longer exist (redirect them), and files.

One honest caveat the tool states out loud: the Search Console "Pages" export is an impressions report, not an index report. A page that is indexed but has no impressions won't appear in it. When that export is the only panel, the step is called "with impressions in search" rather than "in the index", and pages without impressions are not reported as unindexed.

doctor --live separates what is done from what is left. An export shows the past: on one live site four of five "missing" pages already answered 301, and on another six of seven stray hosts were already closed with noindex. With --live each such address is checked over the network — a page is fine if it redirects or answers 404/410, a host if it sends noindex, sits behind a login or has moved to another host — and only what still needs a hand is listed. Redirects within a host are followed to the final answer: a GitLab root that 302s to a public sign-in page is open, not closed. It is the one part of doctor that uses the network, so it runs only when asked.

Multilingual and multi-region

A programmatic pipeline is usually multilingual, and that breaks in ways a single-language site never does.

Text volume and title/description length are measured by each page's script, in display width — not by one language detected for the whole project. That distinction is not academic: on a live 2,970-page catalogue in ten languages the project language came out as en, every Chinese page was counted as English, and all 174 thin findings were false. Width also handles mixed strings, which is what real pages are: those Chinese titles are 43% Han and the rest Latin.

Anchor length is judged only in scripts where length means something. All 914 vague-anchor findings on that same site were false — "यमन" (Yemen) and "হোম" (Home) are three characters and complete words.

hreflang is checked as a cluster: a missing self-reference, a one-way link (Google discards the whole cluster rather than counting it partly), an alternate that is noindex or canonicalised elsewhere, a canonical that leaves the language and cancels the cluster, a missing x-default, and language codes where a country code was meant — uk is Ukrainian, not the United Kingdom. On a monolingual site none of this runs.

And en-us versus en-gb is not a duplicate. Those pages are legitimately near-identical; "keep one and set a canonical" would delete the regional version. Pairs inside one hreflang cluster are excluded from the duplicate check and reported separately.

Content-type profiles

Thresholds differ across content types by substance, not taste. 250 words is normal for a guide and absurd for an event card; fact-checking is meaningless where there is no dataset at all.

stale-event catches an event whose date has passed while the page stays open to indexing. That isn't a traffic problem, it's a trust problem: someone drives to a concert that no longer exists.

For ugc, silence is not a clean bill of health — it means there was nothing to check against, and the tool prints that as a line rather than leaving you to assume.

Portfolio

Separate reports answer "what's wrong with this site". A portfolio answers the question you can't see in them: what breaks the same way everywhere.

  · visa       2933 pages  critical  412  [catalog]
  · events      840 pages  critical   61  [events]
  · feed       5100 pages  critical    0  [ugc]
  · product      34 pages  critical    2  [product]

Shared problems, as a share of each project's pages:
  orphan        3 projects: visa 41%, events 38%, product 35%
  same-opening  2 projects: visa 62%, events 55%

Shares, not counts: a hundred findings across three thousand pages and ten across twenty are the same disease at different volumes. One project failing doesn't stop the run — it becomes a line in the report.

What it does not claim

It does not promise citations in AI search, and it says so in the output. Ahrefs, across 75,000 brands, found AI visibility correlates most with mentions off your site (0.66–0.74) and with page count at 0.19 — which is exactly what a programmatic pipeline produces. 76% of AI Overview citations come from pages already ranking in the classic top 10.

So: machine readability is a necessary condition and this tool's job. Getting cited is decided by work outside your files, and that is not a code problem.

indexgap cite measures the thermometer, and says so: it asks Perplexity, the OpenAI Responses API, the Gemini API and Grok a set of real questions and counts how often your domain comes back in the sources. It reports a share of runs, not yes/no, because these answers are not deterministic. And it never says "ChatGPT cites you" — what it measured is the API, which OpenAI's own docs describe as behaving differently from the product. It is the only command that needs API keys, it is off by default, and it sends nothing without --send.

Where an engine publishes what it actually cited, the thermometer is not needed. Bing Webmaster Tools does, for Copilot, and indexgap doctor reads that export directly — observed citations per page, no keys, no sampling of your own. It still says what the number is not: a count of citations, not a ranking, and silence in it proves nothing about indexing.

No llms.txt generator either. Google has stated it does not support it and has no plans to; no engine has confirmed using it for ranking. Generating a file nobody reads is a ritual, not a feature.

It also doesn't write or rewrite content, check rankings, or call paid services. indexgap brief formulates the task and stops there — on purpose. The central check compares the numbers on a page against the dataset row that produced it; if the package supplied those numbers itself, the check would be checking its own output and would always be green. Rejecting keywords and acting on contested findings is always confirmed by a human.

And it does not replace Search Console — it leads you there. If the tool says one thing and Search Console says another, Search Console is right.

Tests

python3 -m unittest discover -s tests

301 scenarios. Each one is a reproduced defect found by two waves of adversarial review and one run against six live sites, plus the behaviour of profiles, portfolio and project installation. The rule: a finding without a test comes back.

Calibrated on live sites, not fixtures

Thresholds are not guesses. They were checked against six production sites — 7,149 sitemap URLs, 5,041 pages fetched and parsed — and the run changed the tool in three ways:

• One cause, not four findings. Two of the six sites served every page as an empty JavaScript shell. The tool reported 1,099 js-shell and 1,099 low-uniqueness and 1,098 orphan — one disease counted four times. Checks that need text or links are now skipped on a shell, and the run says how many shells there were.
• Duplicates are groups, not pages. 588 near-duplicate pages turned out to be 72 groups, the largest holding 24. "Rewrite 588 pages" is a sentence; "untangle 72 topics" is a task.
• A finding on every page is a template property. vague-anchor fired on 2,970 of 2,970 pages — the culprits were a language switcher (中文) and a social link (VK), short in characters and perfectly informative. Anchor length is now measured in the right unit, and any code that hits ≥90% of pages is labelled as something to fix once in the template.

For agents

SKILL.md at the root is the overview. indexgap/skills/ holds four stage-specific skills — indexgap-plan, indexgap-review, indexgap-publish, indexgap-portfolio — which indexgap init copies into your project's .claude/skills/. A narrow skill fires more accurately than a broad one: the agent sees only the commands and finding codes for the stage it's in.

Codex users: indexgap init --agents writes a marked block into AGENTS.md.

MIT.