Overview

A Dash app is a JavaScript application. Ask for any page without running JavaScript — which is what most crawlers, link previewers and LLM fetchers do — and you get this:

<div id="react-entry-point">
    <div class="_dash-loading">Loading...</div>
</div>

Every URL returns the same empty shell. To a search engine, a 30-page documentation site looks like 30 identical thin pages. To an agent, it looks like nothing at all.

dash-improve-my-llms fixes that at the framework level. You write each page's content once, as Markdown, and the package serves it everywhere a non-JavaScript consumer will look:

One source of truth, eight surfaces, and the interactive app is untouched. Every llms document content-negotiates: agents and crawlers get the raw Markdown byte-for-byte, browsers get the same Markdown rendered (?raw=1 forces the raw side). configure_seo() additionally answers /favicon.ico and the apple-touch paths with a redirect to a declared icon, so Google's fallback path stops receiving the app shell.

Installation

pip install "dash-improve-my-llms[flask]"     # Dash's default backend
pip install "dash-improve-my-llms[fastapi]"   # Dash 4.2+
pip install "dash-improve-my-llms[quart]"     # Dash 4.2+ async
pip install "dash-improve-my-llms[all]"

The backend is detected from app.server; your code is the same either way. Requires dash>=4.1. See Dash compatibility for the tested matrix.

Quick start

import dash
from dash import Dash, html
from dash_improve_my_llms import add_llms_routes, RobotsConfig

app = Dash(__name__, use_pages=True)

# Absolute URLs in sitemap.xml, llms.txt and canonical tags come from this.
app._base_url = "https://myapp.com"
app._robots_config = RobotsConfig(block_ai_training=True)

app.layout = html.Div([dash.page_container])

add_llms_routes(app)   # ← the whole integration

Then give each page prose, either at module scope:

# pages/equipment.py
LLMS_DOC = """
# Equipment Catalog

> Browse, search and filter the equipment inventory.

Filter by category, availability, or location. Selecting a row opens the
detail view with maintenance history. See [the API guide](/api) for
programmatic access.
"""

…or explicitly, which is what you want when pages are generated in a loop:

from dash_improve_my_llms import register_page_metadata

register_page_metadata(
    "/equipment",
    name="Equipment Catalog",
    description="Browse and filter equipment.",
    llms_doc=markdown_text,
)

register_page_metadata merges. Calling it again with only a name will not erase the prose you registered earlier.

At startup the package names every page that has no prose:

UserWarning: dash-improve-my-llms: 4 pages have no LLMS_DOC source
(/changelog, /docs, /privacy, /terms).

Treat that warning as a to-do list. Those pages are the ones serving a stub.

Documentation

Full documentation lives on the package's reference deployment and R&D bench — a separate repository, llms-2plot-dev, built on the network's documentation boilerplate and demonstrating every claim in this README on real URLs:

📚 llms.2plot.dev

Every page there also serves /<page>/llms.txt — the prose as Markdown, ready to paste into a chat window. THIS repository is the package and its test bed, nothing else: the multi-backend suite under tests/ boots real Dash apps against the working tree (pytest tests --ignore=tests/legacy), and scripts/matrix.py reproduces the CI matrix locally.

How the content gets served

Two independent paths, both on by default:

1. Universal prerender. Every visitor's initial HTML carries the page's prose inside #react-entry-point, plus a per-page <title>, meta description, canonical URL, and JSON-LD. Dash's createRoot().render() replaces that block when React mounts, so the live app is unaffected — but anything that reads the HTML without executing it sees real content.

2. Crawler short-circuit. A recognised crawler gets a complete static HTML document before Dash runs at all. Cheaper, and it covers clients that would choke on the app shell entirely.

The prerender is what makes this not cloaking: users and crawlers receive the same content. Google deprecated dynamic rendering as a recommendation in 2022, and a stub-for-users/prose-for-crawlers split is the pattern that gets flagged. Serving real HTML to everyone sidesteps the question and improves LCP at the same time.

To fall back to crawler-only rendering:

add_llms_routes(app, LLMSConfig(prerender=False))

llms.txt is a document, not a dead end

A page's llms.txt gets fetched in isolation — pasted into a chat, handed to an agent. On its own it says everything about that page and nothing about where it sits, so there is nothing to follow and exploration stops. Each one therefore opens with three lines of orientation:

# FastAPI Showcase

> What the FastAPI backend unlocks in this boilerplate.

**Site index:** [https://docs.example.com/llms.txt](…) — every page on this site, as Markdown.
**Network index:** [https://hub.example.com/llms.txt](…) — The Example Network; start here to discover sibling sites.
**Sitemap:** https://docs.example.com/sitemap.xml

Disable with LLMSConfig(llms_nav=False).

The same URL, rendered, for people

Paste that URL into a browser and you get the document rendered behind a header carrying the network identity — and, if you point the app at a bulletin endpoint, centrally-published tips and announcements that one host controls for the whole network.

Agents, crawlers, curl and link unfurlers receive the Markdown byte for byte. The header exists only in the HTML variant, so it costs an agent nothing — which is why it isn't simply prepended to the Markdown for everyone.

Accept: */*                       → text/markdown
Accept: text/html (real browser)  → text/html, rendered
Accept: text/html (Googlebot UA)  → text/markdown
?raw=1                            → text/markdown, always
?format=html                      → text/html, always

Every variant sends Vary: Accept, and the rendered view is noindex so it never competes with the real page. Disable with LLMSConfig(llms_viewer=False).

Multi-domain networks

Sitemaps are scoped to one origin, so a network spread across hosts has no crawl graph between them. An agent that lands on maps.example.com has no path to email.example.com; a model reasoning about eighteen packages sees one.

Declare the relationships explicitly and they appear in /llms.txt, in the prerendered HTML, and as <link rel="related">:

from dash_improve_my_llms import register_network

register_network(
    name="The Example Network",
    description="A family of open-source Dash component libraries.",
    hub_url="https://hub.example.com",    # one host an agent can enumerate from

    # Same network, same operator.
    peers=[
        {"name": "Hub", "url": "https://hub.example.com",
         "description": "Network index and account origin."},
        {"name": "Maps", "url": "https://maps.example.com",
         "description": "Mapping components for Dash."},
    ],

    # Yours, on its own unrelated domain.
    affiliated=[
        {"name": "Side Project", "url": "https://sideproject.example",
         "description": "A separate product built on the same stack."},
    ],

    # Third-party docs you depend on. Emitted with rel="nofollow".
    external=[
        {"name": "Dash Mantine Components",
         "url": "https://www.dash-mantine-components.com",
         "description": "UI layer these docs are built with."},
    ],
)

The three tiers stay distinct on purpose. Flattening them would either overclaim ownership of third-party sites or bury your own network in an undifferentiated link list — and both make the directory less useful to whatever is reading it.

See docs/NETWORKS.md for the full multi-domain guide and docs/BULLETIN.md for the shared tips/announcements contract.

Access control

Static hiding first — excluded from every surface, 404 to crawlers, never prerendered, absent from MCP:

from dash_improve_my_llms import mark_hidden

mark_hidden("/admin")

For applications with accounts, configure_access decides per request who may read a document, across every surface the package owns:

from dash_improve_my_llms import configure_access

configure_access(
    check,                    # (path) -> "allow" | "gated" | "deny"
    gate_doc=my_gate_doc,     # Markdown served instead of gated prose (200)
    link_suffix=my_suffix,    # authority carried in generated links
)

• allow serves the prose. gated serves the gate document at 200 and keeps the URL listed — the page's existence is public, its content is not. deny is a 404 and delisted, indistinguishable from a page that never existed.
• A check that raises degrades to gated — a bug can neither publish gated prose nor black-hole the site.
• link_suffix lets a signed-in user's copied URL carry its own authority, so the one consumer these documents exist for — an agent fetching with no cookie — still gets in.
• configure_viewer_identity(provider) renders the signed-in reader in the HTML viewer's header; agents receive byte-identical Markdown without it. Identity- or authority-bearing responses send Cache-Control: private, no-store so no CDN can hand one visitor's document to the next.

The full contract is in docs/ACCESS.md.

Who is reading, and what they read

Every document this package serves can be accounted for: which vendor asked, whether that identity checks out, which document they got, and how many bytes went out.

from dash_improve_my_llms import on_document_read

@on_document_read
def record(event: dict) -> None:
    # one call per document served — the package does no I/O with it
    db.insert(event)

The event names vendor_key, bot_type, verified, policy, tier, lane, verdict, status, bytes, path, host, ua, client_ip and ts. Every key is always present; None means "not known here". policy is the exception: since 2.9.0 it is always one of allow, meter or block — the posture the read was actually served under, so a rollup keyed on (vendor, verified, policy) has a third key to group by. Callbacks run on the request path, so append to a queue rather than blocking — and a callback that raises is caught and warned about once, never allowed to take a document down.

verified is not a gate. For the eleven registry vendors whose operators publish their crawler IP ranges — Google, OpenAI, Microsoft, Apple, Perplexity, DuckDuckGo, Common Crawl — the client address is checked against the published list and the event says verified or unverified. For everyone else it says n/a; Anthropic, notably, publishes no ranges, so ClaudeBot is unverifiable and is recorded as such rather than guessed at. An unverified client is served exactly what a verified one is served. The point is a row that says so, not a wall. The ranges ship with the package and are refreshed at release time by scripts/refresh_ip_ranges.py; nothing is fetched on the request path.

One classifier, not four

If your application keeps its own list of bot User-agents, replace it with the registry this package already maintains:

from dash_improve_my_llms import classify

classify(request.headers.get("User-Agent", ""), client_ip=ip)
# {'bot_type': 'training', 'vendor_key': 'gptbot', 'vendor_class': 'training',
#  'verified': 'n/a', 'lane': 'crawler'}
#
# bot_type is one of: training | search | traditional | monitor | unknown
# (monitor — uptime probes and headless automation — is new in 2.9.0)

This is the classification entry point. Hand-written lists drift — they keep retired tokens alive and miss new vendors — and a drifted list is how bot traffic gets mis-attributed in dashboards. The older predicates (is_ai_training_bot, get_bot_type, ...) still work and delegate here.

Dash compatibility

Verified against every supported Dash release on all three backends — scripts/matrix.py reproduces the CI matrix locally.

¹ Dash 4.1.0 has no pluggable backends; Dash(backend=...) does not exist.

² Dash 4.3.0's FastAPI backend is broken upstream. Its page catch-all does not call set_current_request(), so every non-root URL raises RuntimeError: No active request in context and returns a 500. This reproduces on a stock Dash app with this package uninstalled, and is fixed in Dash 4.4.0. The package warns at startup if it detects the combination. Use dash>=4.4.0, or run Flask/Quart on 4.3.0.

Markdown support

LLMS_DOC is rendered by a dependency-free subset renderer covering headings, paragraphs, links, fenced code, images, blockquotes, ordered and unordered lists, pipe tables, horizontal rules, and inline code/bold/italic/strikethrough.

Links matter most: they are what gives your prose an internal crawl graph. [the API guide](/api) becomes a real <a href="/api"> in the served HTML.

All content is escaped before any tag is emitted, link targets are checked against a scheme allowlist, and JSON-LD is \u-escaped so author-supplied page names cannot break out of their <script> block.

rST-style directives (.. toc::, .. llms_copy::) are stripped from prose but preserved inside code fences, so a page documenting them still renders its examples.

API reference

Development

pip install -e ".[dev]"

pytest tests --ignore=tests/legacy          # unit + adapter tests
python scripts/smoke_test.py                # boot a real app, all backends
python scripts/matrix.py                    # every Dash version
python scripts/matrix.py --pytest           # ...and the full suite in each

CI runs the same matrix on every push. Releases publish to PyPI from a version tag via trusted publishing; see .github/workflows/cd.yml.

What this package deliberately does not do

It does not introspect layouts or callbacks. Dash 4.3's MCP server exposes that live and structured, and duplicating it means maintaining a worse copy. mark_important() and mark_component_hidden() are deprecated no-ops — put emphasis in your LLMS_DOC instead.

Community & support

Come build with us:

• 💬 Discord — discord.gg/WEnZR35mrK
• ▶️ YouTube — @2plotai
• 🐛 Issues — github.com/pip-install-python/dash-improve-my-ai/issues

More from Pip Install Python LLC

dash-improve-my-llms is one of several tools built and maintained by Pip Install Python LLC:

License

MIT — see LICENSE. Built by Pip Install Python LLC to make Dash apps first-class citizens of the agentic web.