coop-data-doc

Install: pipx install coop-data-doc — see Install for the one-time pipx setup.

Part of the Cooptimize coop suite — alongside coop-sql-review and coop-dax-review. If your team uses coop-agent, coop install installs all three tools at once and coop update keeps them current.

Automatic documentation for your data estate. Point this tool at whatever source you have today — SQL, Power BI, selected folders from either, both, or neither yet — and it maps how data flows through everything available:

silver table → stored proc → gold table → view → semantic model → measure → report

…then writes two kinds of documentation from that map:

• 📄 Markdown files with machine-readable headers — built for AI agents and scripts (see For AI agents below)
• 🌐 A searchable website with dark mode and clickable, collapsible lineage trees — built for humans. It works straight off your hard drive: no server, no internet, no login.

Everything runs on your machine by reading files. It never connects to a database, never calls an AI service, and produces byte-identical output for identical inputs — so the generated docs can live in git and be checked in CI.

Table of contents

1. Before you start
2. Install
3. First run — about 5 minutes
4. Day-to-day use
5. Compose review findings into the portal
6. The config file, explained
7. When it asks you questions
8. Adding your own notes to the docs
9. Command reference
10. Keeping the tool updated
11. Using it in CI
12. 🤖 For AI agents
13. Troubleshooting
14. Notes on .pbix files

Before you start

You'll need three things:

1. A terminal. That's the text window where you type commands.
   • macOS: press Cmd+Space, type Terminal, press Enter.
   • Windows: press the Windows key, type PowerShell, press Enter.
   • Linux: any terminal; the macOS commands apply as-is.
2. Python 3.10 or newer. Check by typing python3 --version (macOS) or python --version (Windows) and pressing Enter. If that prints something like Python 3.12.4, you're set. Otherwise install Python from python.org first.
3. Any source folders you have on your machine — SQL-only, Power-BI-only, selected folders, both sides, or none yet (discovery mode) are supported. ("Repo" = a folder managed by git; "cloned" = downloaded to your machine with git or GitHub Desktop.) You'll need their folder paths — to get a path, drag the folder onto the terminal window (macOS), or right-click it in File Explorer and choose Copy as path (Windows).

💡 Terminal survival kit: you type a command, press Enter, and read what comes back. cd some/folder moves you into a folder. Paste with Cmd+V (macOS) or right-click (Windows). In menus: ↑↓ to move, Enter to select, Ctrl+C to safely cancel.

⚠️ Windows users: wherever this README shows python3, type python instead — that applies to every command below.

Install

Use pipx, not pip install. pipx puts the tool in its own isolated environment, so it can never upgrade or downgrade packages that other tools (e.g. ms-fabric-cli, azure-cli) depend on. Installing into your system Python with plain pip works but will fight those tools over shared dependencies like pyyaml — see Troubleshooting if you've already hit that. One-time pipx setup:

python3 -m pip install --user pipx      # Windows: python -m pip install --user pipx
python3 -m pipx ensurepath              # Windows: python -m pipx ensurepath

Close and reopen your terminal, then install the tool from PyPI:

pipx install coop-data-doc

That's it. (Advanced — install a specific unreleased commit straight from the source repo instead: pipx install git+https://github.com/kabukisensei/coop-data-doc.git.)

Check it worked:

coop-data-doc --version

You should see coop-data-doc, version X.Y.Z matching the latest release on PyPI. If the terminal says "command not found" (macOS) or "the term 'coop-data-doc' is not recognized…" (Windows), the install location just isn't on your PATH yet — see Troubleshooting to fix it permanently.

Always-works fallback: every coop-data-doc … command also runs as python3 -m coop_data_doc … (Windows: python -m coop_data_doc …). This invokes the exact same tool through Python directly, so it works even before the PATH is fixed — e.g. python3 -m coop_data_doc --version or python3 -m coop_data_doc setup.

uv tool install /path/to/coop-data-doc     # if you use uv
pip install /path/to/coop-data-doc         # plain pip, into the current environment
pip install -e "/path/to/coop-data-doc[dev]"   # contributors: editable + test deps

First run — about 5 minutes

Step 1 — make a home for the docs. Create a folder for the generated documentation near any source folders you have. If you have no local source yet, create the docs home anyway and choose discovery mode. For example:

cd ~/repos          # Windows: cd C:\repos
mkdir my-data-docs
cd my-data-docs

Step 2 — run the tool with no arguments:

coop-data-doc

Because there's no configuration here yet, it offers to walk you through setup. Choose "Set up interactively" (↑↓ + Enter) and answer the questions:

• Project name — the title shown on the docs website.
• Available local source — choose both, SQL only, Power BI only, or discovery mode. The wizard asks paths only for sources you selected and can be rerun later.
• Which folders to document — once a repo path is set, the wizard lists that repo's top-level folders as a checkbox. Nothing starts checked — folder selection is an allowlist: only the folders you pick are crawled. Press Space to check each folder you want documented and Enter to confirm (an empty selection is rejected and re-asked). No need to type patterns by hand. (If the repo isn't on disk yet, it falls back to asking for include globs as text.) Each checked folder becomes file-type-scoped entries under repos.<key>.include (e.g. procs/**/*.sql); nested exclude patterns you've added by hand are kept as-is on a re-run.
• Which semantic models to document (Power BI repo) — the wizard finds every *.SemanticModel folder and shows them as a checkbox. Pick the ones you want; only those are crawled (their reports come along automatically, scoped to the folders you picked above, and .pbip/other loose files are left out). You don't need the .pbip file — everything is read from the .SemanticModel/definition/ TMDL.
• Which schemas to document, layer by layer — the wizard lists every schema its scan discovered under each medallion layer (bronze/silver/gold). A checked schema is both included in the docs and assigned that layer; an unchecked schema is left out entirely. A final "include without a layer" checkbox offers whatever is left over, to document it with automatic layer detection. The checked schemas are written to layers.<layer>.schemas and their union to include_schemas.
• Output folders — press Enter to accept the defaults.
• Connecting Power BI tables to their SQL sources — if both sides are on disk, the wizard does a quick read-only scan and reports how many Power BI tables already link to a SQL object automatically. For any that don't, it works out which SQL schema the table's name actually lives in and asks you to confirm a one-line mapping (e.g. "Map Sales Analytics → mart?") — no typing schema names blind. A table whose name matches no SQL object is left unresolved rather than guessed, and a re-scan confirms the mappings took. (When the repos aren't cloned yet, it falls back to asking for mappings as text.)

Step 3 — build the docs. Run the bare command again and choose "Update the docs" — or run coop-data-doc update. (If the tool suggests coop-data-doc build: build and update are the same command.) You'll see a summary like:

Warnings:
  dynamic_sql                    1
  ...
26 objects, 30 lineage edges (2 cross-repo links; 1 unresolved)
Markdown docs: /Users/you/repos/my-data-docs/data-docs
HTML portal:   file:///Users/you/repos/my-data-docs/data-docs-site/index.html

A Warnings block is normal and informational — the build succeeded if you see the object count and the two output paths. (Each warning category is explained in Troubleshooting.) The first run may also ask a few mapping questions — see When it asks you questions.

Step 4 — open the website. Copy the file://… line into your browser's address bar — or:

open data-docs-site/index.html        # macOS
start data-docs-site\index.html       # Windows
xdg-open data-docs-site/index.html    # Linux

You'll get a searchable site with a page per table, view, stored procedure, semantic model, measure, and report — each with its defining SQL/DAX up top, its columns, where its data comes from, and what depends on it (upstream/downstream, each entry linking to its own page). Model-facing pages add extras: a collapsible "trace back to source" tree that walks a measure or gold table all the way down to its bronze sources, Relationship Grid (a fact × dimension matrix) on each semantic model, an Unused measures roll-up for cleanup, and reports nested under the model they draw from.

Step 5 (recommended) — commit everything. One command at a time:

git init
git add -A
git commit -m "Initial data docs"

If this is your first-ever git commit on this machine, git may ask you to introduce yourself first: git config --global user.name "Your Name" and git config --global user.email "you@example.com" — then re-run the commit.

Day-to-day use

When SQL or Power BI changes land in your repos, refresh the docs:

cd my-data-docs
coop-data-doc            # choose "Update the docs"  — or —
coop-data-doc update     # the same thing, no menu

Pages for new objects appear, changed objects update, and pages for deleted objects are removed. Notes you've written in Business Intent blocks are preserved. Re-running is always safe.

Compose review findings into the portal

If your team also runs the suite's linters — coop-sql-review and coop-dax-review — the portal can show their findings on the object pages they're about, so "which gold tables feeding this model have open SQL findings?" is answered in one place. First produce the machine-readable reports:

coop-sql-review check sql-repo --format json -o sql-findings.json
coop-dax-review check pbi-repo --format json -o dax-findings.json

…then hand them to the build (the flag is repeatable, any number of files):

coop-data-doc build --reviews sql-findings.json --reviews dax-findings.json

Or list them once in coop-data-doc.yml (paths relative to the config file; the --reviews flag extends the list):

reviews: ["sql-findings.json", "dax-findings.json"]

What you get, all rendered at build time (the lineage graph — graph.json / manifest.json — is untouched):

• Object pages — a matched object gains a ⚠ N review findings chip under its title and a "Review findings" table (severity, rule, message, file:line).
• A top-level Findings page — summary counts per tool and severity, every finding grouped severity → rule with links to the object pages, and a "Not matched to a documented object" section so findings about undocumented objects are listed, never dropped. (agent_review judgment items are counted there but not joined — v1 composes findings only.)
• Matching uses the same normalized schema.name identity as the lineage linker: SQL findings attach to SQL objects (tables/views/procs), DAX findings to the measure or table in their model (a model-level finding attaches to the model's page).

Two things to know:

• Advisory, always. Findings never change exit codes — build --strict and check gate on the same things they always did. An unreadable/foreign review file degrades to a reviews_unreadable warning (the build succeeds); a --reviews path that doesn't exist at all is a usage error (exit 2), like any bad argument.
• check needs the same inputs. Review files are an explicit build input, so a CI coop-data-doc check must pass the same --reviews arguments the build used (the config's reviews: list is picked up automatically) — otherwise the trees legitimately differ and check reports the docs stale.

The config file, explained

Setup writes a single file, coop-data-doc.yml, which you can edit by hand anytime (or re-run coop-data-doc setup — it pre-fills your current answers so you can change just one thing):

project_name: Coop BI Estate        # the title shown on the docs website

repos:
  sql:                              # your SQL repo
    path: ../sql-repo               # relative paths start from THIS file's folder
    include: ["**/*.sql"]           # which files to read
    exclude: ["**/archive/**"]      # which files to skip (wins over include)
  powerbi:                          # your Power BI repo (PBIP/TMDL + PBIR)
    path: ../pbi-repo
    include: ["**/*.tmdl", "**/*.bim", "**/report.json", "**/visual.json", "**/page.json", "**/*.pbix"]
    exclude: []

# Discovery mode is simply `repos: {}`. Add any source labels later; `sql` and
# `powerbi` are conventional names used by the wizard, not required engine keys.

schema_mappings:                    # hint: which view schema feeds which model
  - schema: sales
    model: "Sales Analytics"

layers:                             # medallion layers (all optional)
  bronze:
    schemas: [erp_orders, erp_finance]   # source schemas
    paths: []
  silver:
    schemas: [stg]
    paths: []
  gold:
    schemas: [mart, common]         # the proc schema + shared/common schema
    paths: ["**/dim/**", "**/fact/**"]   # gold table folders

ignore_schemas: [staging, scratch]  # schemas to drop entirely (never documented)
include_schemas: [erp_orders, erp_finance, stg, mart, common]
                                    # optional: when non-empty, ONLY these schemas
                                    # are documented (the wizard writes the union of
                                    # the schemas you check in its layer questions)

# reviews: ["sql-findings.json"]   # optional: coop-sql-review / coop-dax-review
                                    # --format json reports to compose into the
                                    # portal (see "Compose review findings" above).
                                    # setup only writes this block when non-empty.

branding:                           # optional HTML-site branding (all optional)
  logo: ./assets/logo.png           # paths resolve against THIS file's folder
  favicon: ./assets/favicon.ico
  primary_color: "#004060"          # header / nav / links (defaults to this)
  accent_color: "#e04020"           # hover / active (defaults to this)

output:
  dir: ./data-docs                  # the markdown (agents read this)
  site_dir: ./data-docs-site        # the website (humans read this) — must be a
                                    # SEPARATE folder, not inside dir (see note below)

sql_dialect: tsql                   # covers SQL Server, Azure SQL, Fabric warehouse

schema_mappings matters because view schemas and semantic-model names are often similar but not identical — e.g. the sales schema feeds the "Sales Analytics" model. Each hint you add means fewer questions on the next run.

Medallion layers (bronze / silver / gold)

The object type (table / view / stored proc) is detected automatically from the SQL. The layer can't be — a CREATE TABLE doesn't say "I'm silver" — so you declare it. A table or view is assigned the first layer (precedence gold → silver → bronze) whose:

• schemas list contains its schema, or
• paths globs match its file.

In a Fabric or SQL warehouse the folder layout is usually <Warehouse>/<schema>/<ObjectType>/, so the schema is the folder — listing schemas is all you need, and you can ignore paths entirely. paths exists only for the less-common case where a layer maps to a folder that isn't its own schema (say a dim//fact/ convention living under another schema). You can mix both — a node is assigned the first layer (gold → silver → bronze) hit by either its schema or its path — but most estates only ever fill in schemas. The setup wizard reflects this: it asks for schemas layer-by-layer (a checked schema is both included in the docs and assigned that layer — unchecked schemas are excluded) and only asks about folders if you opt into the "advanced" question.

Each layer is optional — omit bronze or silver entirely to skip it. Anything no rule matches falls back to a read/write heuristic (a table that's only ever read → silver; one that's created here → gold), and the scan warns which objects fell back so you can add rules. Bronze only ever appears when you declare it.

Choosing which schemas get documented at all: when include_schemas is non-empty, only those schemas are documented — every other schema's objects are dropped (the wizard writes it as the union of the schemas you check in the layer questions, so wizard-produced configs are strictly opt-in). Empty means no restriction, which keeps older hand-written configs working unchanged.

Dropping schemas you don't want documented: list them in ignore_schemas (the wizard asks for these too; it wins over include_schemas on conflict). System schemas — sys, information_schema, tempdb, guest, db_* — are always dropped automatically, since they're catalog references, not real data. Note that ignoring a schema removes it from lineage entirely, so anything downstream loses that upstream link.

Full configuration reference

include / exclude — choosing what gets crawled

Patterns are matched against each file's path relative to its repo, forward-slashed: **/Foo/** = anything under a folder named Foo; **/*.sql = any .sql anywhere; SomeDir/** = everything under SomeDir. Foo/**/*.sql also matches files directly under Foo.

The setup wizard (and set-folders) writes allowlists. Picking a top-level folder writes file-type-scoped include globs for it — **/*.sql becomes Foo/**/*.sql — so only the picked folders are crawled. A repo whose include list contains such folder-scoped globs is in allowlist mode; picking the schemas in the layer questions additionally writes include_schemas, so only those schemas are documented. The definition.pbir always-include carve-out is disabled in allowlist mode, so report metadata can't leak in from an unpicked folder.

Hand-written configs can use either strategy, and older denylist configs keep working unchanged:

• Allowlist (cleanest for big repos): include only real object folders, e.g. include: ["**/Tables/*.sql", "**/Views/*.sql", "**/StoredProcedures/*.sql", "**/Functions/*.sql"] — everything else (deployment scripts, role grants, notebooks) is ignored automatically.
• Denylist (legacy): keep include: ["**/*.sql"] and drop noise with exclude: ["**/logging/**", "**/Deployment/**", "**/BACKUP/**"]. Re-running the wizard over such a config starts with every folder unchecked — pick the folders to keep and it converts the repo to the allowlist form.

Use coop-data-doc scan (fast, no rendering) as the feedback loop: watch the object count and the diagnostics summary, adjust, repeat. Reports need to be present in PBIR format (a committed .Report/definition/pages/.../visual.json tree) or legacy report.json — the default Power BI include already matches them, so once they're in the repo they're picked up with no config change.

Worked example: a large multi-schema warehouse

A config for a Fabric warehouse + Power BI estate with medallion schemas, ERP source schemas, model-named gold schemas, a common schema feeding every model, and editor/backup noise to drop:

project_name: Acme BI Estate
repos:
  sql:
    path: ../fabric-dw
    include: ["**/*.sql"]
    exclude: ["**/logging/**", "**/Security/**", "**/Deployment/**"]
  powerbi:
    path: ../fabric
    include: ["**/*.tmdl", "**/*.bim", "**/visual.json", "**/page.json", "**/report.json"]
    exclude: ["**/BACKUP/**", "**/Documentation/**", "**/Editor and Theme Files/**"]
layers:
  bronze:
    schemas: [dbo]                    # lakehouse landing tables
  silver:
    schemas: [erp_orders, erp_finance]   # ERP source schemas
  gold:
    schemas: [silver, mart, common, sales, ops]  # 'silver' schema is gold here!
    paths: ["**/dim/**", "**/fact/**"]
ignore_schemas: [staging, scratch, sandbox, legacy]

The standout: a schema named silver can sit in the gold layer — assignment follows your rule, not the schema's name.

When it asks you questions

Most cross-repo links resolve on their own (an exact schema-and-name match needs no configuration). setup proposes the few schema mappings that are genuinely needed up front — derived from a dry-run scan, so you confirm rather than type (see the first-run walkthrough above). With those in place, a build rarely needs to ask anything.

When a Power BI table's source still can't be matched to a SQL object — names are close but not identical — the build shows a pick-list: the most likely candidates with similarity scores, plus "Mark as external source" (for data that doesn't live in these repos) and "Skip for now".

Every answer — including skips — is saved instantly to .lineage-cache.json, which lives next to coop-data-doc.yml (it's a hidden file; git add -A picks it up even if Finder/Explorer doesn't show it). Commit that file. It's what makes every later run — yours, a coworker's, CI's — fully automatic.

Two things worth knowing:

• If you cancel mid-way (Ctrl+C), the answers you already gave are kept; run again to continue from where you stopped.
• "Skip for now" is remembered too, so you won't be re-asked on the next run. To be asked again (or change any answer), open .lineage-cache.json in a text editor and delete that entry, then re-run.

Adding your own notes to the docs

Every generated page has a Business Intent section between two marker comments:

## Business Intent

<!-- intent:begin -->
Write anything here: what this table is for, who owns it, gotchas.
<!-- intent:end -->

Text between the markers survives every rebuild verbatim. Everything outside the markers is regenerated, so put your notes inside them.

Command reference

Config discovery: coop-data-doc searches for coop-data-doc.yml in the current directory and walks up parent directories (like git finding .git). You can override with --config PATH (which takes precedence) or the COOP_DATA_DOC_CONFIG environment variable. A non-empty environment value is authoritative: missing, directory, unreadable, or invalid paths fail rather than falling back to another config. Relative environment paths resolve against the process current working directory. Leave the variable unset or empty to use normal upward discovery. config-set, setup, and init may create an authoritative missing target; show-config reports it as exists: false.

Options for build/update: --skip-html (markdown only), --serve (live-preview the site). scan/build/update all accept --non-interactive (never prompt; for CI) and --strict (exit code 2 on unresolved references or risky/unresolved warnings — the same categories check gates on). scan, build, update, and check also accept --no-parse-cache — force a cold SQL parse, bypassing the incremental per-file parse cache (see below). scan/build/update accept --jobs N to parallelize SQL parsing across processes (see below). Every pipeline command accepts --config PATH (default: discover in cwd and parents). Global flags go before the subcommand: --version, -v (debug + tracebacks), -q (quiet), and --log-file PATH (write a verbose debug log to a file, leaving the console at warning level) — e.g. coop-data-doc -q update or coop-data-doc --log-file build.log build.

Parallel SQL parsing (--jobs N). scan/build/update parse each SQL file across a process pool, defaulting to your CPU count (capped at 8). It is transparent and deterministic: the per-file results are merged back in sorted file order, so a --jobs N build is byte-for-byte identical to a --jobs 1 build and to a cold serial build — the same determinism guarantee the parse cache already meets, warm or cold. Only the SQL parsers fan out; every cross-file pass stays serial. --jobs 1 is the exact sequential path, and tiny estates parse serially automatically (the pool's startup cost would outweigh the win). Set a default without the flag via COOP_DATA_DOC_JOBS (e.g. COOP_DATA_DOC_JOBS=1 for pool-free runs). The worker is spawn-safe, so it works on Windows; a file whose worker fails degrades to a parse_worker_error diagnostic and the build completes, and any failure to start the pool falls back to serial. The speedup scales with per-file parse cost — proc-heavy estates benefit most (a 200-proc corpus parses ~3x faster at --jobs 4); trivial table DDL parses too fast for it to matter.

Incremental parse cache. Each run writes .coop-data-doc-parse-cache.json next to your config: a content-hashed, per-file cache so an unchanged SQL file is not re-parsed on the next build (the console prints e.g. Parsing SQL (312 cached, 5 parsed)). It is fully transparent — a warm build is byte-identical to a cold one — and derivable, so it is gitignored and never committed. It self-heals: a corrupt or version-mismatched cache degrades to a cold parse (a parse_cache_invalid warning, never a failure) and is rewritten. Pass --no-parse-cache to force a cold parse.

Agent / CI commands

Beyond the commands above, the CLI exposes a non-interactive surface for agents and CI (every one emits sorted/deterministic JSON or writes the config, and takes --config PATH):

See AGENTS.md for the full machine-readable contract (flags, exit codes, JSON shapes).

scan/build/update show progress bars on stderr while they work, but only in an interactive terminal — they're suppressed by -q and absent in CI or piped output, and they never affect the generated files.

Keeping the tool updated

coop-data-doc upgrade            # check for a newer release; print the exact upgrade command

upgrade does not self-update — replacing the tool while it's running is unreliable (and impossible on Windows, which locks a running executable). Instead it checks PyPI for a newer release, then detects how this copy was installed (pipx / uv / pip / a git checkout) and prints the exact command to run yourself from a normal shell — e.g.:

pipx upgrade coop-data-doc      # run in a regular terminal, where the tool isn't running

It also lists any out-of-date direct dependencies (flagging major-version jumps for you to review), but it never pulls, reinstalls, or updates anything itself — you copy and run the printed command. This is the single command that touches the internet; documentation builds are always fully offline.

After running the printed command, coop-data-doc --version should report the new version. If it still shows the old number, force a clean re-pull: pipx reinstall coop-data-doc.

Using it in CI

Two useful gates for a pipeline (e.g. GitHub Actions / Azure DevOps):

coop-data-doc check              # fails if committed docs are stale,
                                 # references are unresolved, a file is
                                 # corrupt/undecodable (objects missing), or
                                 # risky/unresolved warnings exist (regex_fallback,
                                 # dynamic_sql, unresolved_partition_source,
                                 # ambiguous_visual_binding — use --lenient to
                                 # tolerate the known-and-accepted ones, but NOT
                                 # a corrupt file, which is never "accepted")

coop-data-doc build --non-interactive --strict   # rebuild; exit 2 on problems

Exit codes: 0 success · 1 stale docs / friendly error · 2 unresolved references, risky/unresolved warnings, error-severity diagnostics (a corrupt/undecodable file or truncated proc — whole objects are silently missing), or an invalid command line (typo'd flag/command) · 130 cancelled with Ctrl+C.

🤖 For AI agents

For the full machine-readable contract, see AGENTS.md. It covers the JSON schema, CLI flags, exit codes, config discovery, and Python API.

The Markdown output (output.dir, default data-docs/) is designed to be read by LLM agents without custom tooling. Here's the quick summary:

Entry points

• data-docs/manifest.json — the entire lineage graph in one JSON file. Best for programmatic traversal and impact analysis. (data-docs/graph.json is a byte-identical copy written by every pipeline run; read manifest.json.)
• data-docs/<type>/<slug>.md — one page per object (its exact location is the path field in each page's front-matter — see Page paths below). Best for reading context about a specific object. data-docs/index.md lists object counts and unresolved items.

Identifiers. Node ids are stable, lowercase strings: "<type>:<schema>.<name>" — e.g. view:sales.dim_customer. Caveats: the <schema>. part is omitted for objects that have no schema (report:sales, semantic_model:sales), and names may contain spaces (measure:sales.total sales). Prefer reading the explicit name/schema fields over parsing ids.

Page paths. Don't compute these — read the path field in a node's front-matter (and resolve ids to pages the same way). The on-disk slug is sanitised for cross-platform safety (Windows-illegal characters removed) and carries a short hash suffix to guarantee uniqueness, so it is not trivially derivable from the id. Every page is at data-docs/<path>.

Page front-matter — strict YAML, fixed key order, all strings double-quoted, non-empty lists in block style (empty lists render as []):

---
id: "view:sales.dim_customer"
type: "view"                              # bronze_table | silver_table | gold_table | view |
                                          # stored_proc | semantic_model | pbi_table | measure | report
                                          # (report pages/visuals fold into the report)
name: "dim_customer"
schema: "sales"                         # SQL schema; for pbi_table/measure nodes it's the
                                          # (lowercased) model name; "" for report/semantic_model
layer: "gold"                             # bronze | silver | gold (from config.layers rules); "" if none
source_file: "views/sales/dim_customer.sql"   # repo-relative; cite this as evidence
path: "view/sales-dim_customer-<hash>.md"     # this page's location under data-docs/ (read it, don't compute it)
upstream_inputs:                          # direct (depth-1) data sources, flow-normalized
  - "gold_table:dbo.fact_sales"
downstream_dependents:                    # direct (depth-1) consumers
  - "pbi_table:sales.dim_customer"
tags:
  - "sales"
---

To go from an id (e.g. one listed in upstream_inputs) to its page, open that node's page via its own path field — or scan manifest.json for the node and read its data directly. Avoid string-building the filename.

Manifest shape. manifest.json has nodes (object keyed by id) and edges (list). Node fields use the internal names node_type and schema_name (the front-matter keys type/schema are renderer aliases for them), plus name, source_file, columns, and metadata. Edge fields: source_id, target_id, edge_type, evidence — edges carry no metadata; trust markers live on nodes.

Traversal rules

• "What breaks if X changes?" → follow downstream_dependents page to page. "Where does this number come from?" → follow upstream_inputs.
• In manifest.json, edges are stored in authoring direction; convert to data-flow direction with this rule: for edge_type ∈ {reads, references, visualizes} data flows target → source; for {writes, feeds, defines} data flows source → target. (Front-matter lists are already flow-normalized — prefer them when reading pages.)
• Column contracts (name, type, nullability, constraints) are in each page's Structural Contract table; measure DAX is on measure pages under DAX.

Trust markers — these live in nodes[<id>].metadata in manifest.json (check the endpoint nodes of an edge; pages don't carry them, and an empty upstream_inputs on a page does not mean the object was verified to have no sources):

Editing: agents may write inside <!-- intent:begin -->…<!-- intent:end --> blocks (those survive rebuilds). Never edit generated content outside the markers — it's overwritten on the next update. To regenerate after source changes, run coop-data-doc update --non-interactive and check the exit code.

Working on this tool's own source code instead? Read CLAUDE.md and ARCHITECTURE.md in the repo root.

Lineage identity and export safeguards

Lineage identity and mapping safeguards: duplicate semantic-model names in distinct source scopes raise identity_collision and stop generation publication. Existing IDs, authored intent and committed mappings remain untouched pending an explicit scope-ID migration. Report byPath bindings require the exact repository/path scope. SQL/M partitions accumulate their observed dependencies; SQL set operations and CTAS retain every table dependency, with unsupported column lineage marked unresolved. Mapping decisions can carry a source_signature from resolve; a changed source requires explicit review even when the previous target still exists. Unsigned legacy (pre-1.3.0) decisions are applied and marked unverified. When the previous published graph shows their source has since changed, they raise cache_source_changed instead, and that review need is recorded in the graph so later builds keep asking until a human answers. A saved decision whose key no current source uses (for example one table-level answer for a table now keyed per partition) is kept on disk, never applied, and named by cache_key_unmatched. resolve exposes changed-source review choices; preserve its signature in resolve-apply.

Stakeholder CSV exports neutralize formula-like source values for spreadsheet opening. JSONL setup always uses UTF-8 for both input and output, including redirected non-UTF-8 Python streams. Native Windows terminal qualification is still required; portable fixtures do not assert Windows runtime acceptance.

Troubleshooting

Notes on .pbix files

.pbix support is best-effort: report layout and Power Query (M) source usually extract; the compiled data model does not. For full lineage, open the file in Power BI Desktop and save as a .pbip project — the git-friendly format these repos should hold anyway. The tool tells you when it hits an opaque model.

Third-party assets

The package vendors iframe-worker 1.0.4 (MIT) so generated sites search over file:// with no network, plus two small first-party scripts — doc-tree.js (collapsible lineage trees) and nav-collapse.js (desktop sidebar-collapse toggle) — hand-rolled rather than vendored to stay within the no-CDN rule. See src/coop_data_doc/templates/assets/README.md for provenance.

Development

pip install -e ".[dev]"
pytest

See CONTRIBUTING.md for the module map and design rules, and ARCHITECTURE.md for how it all works.

License

MIT — see LICENSE.