sqldoc 1.6.0

Multi-database documentation generator + PII/compliance scanner for SQL Server, PostgreSQL, MySQL and Azure SQL, with optional AI-written descriptions.

sqldoc

A seven-command database platform for SQL Server, PostgreSQL, MySQL, SQLite, Snowflake, and Azure SQL — documentation, PII/compliance scanning, health analysis, data-quality profiling, schema intelligence, AI insights, and compliance reporting. One CLI, self-contained HTML reports, and machine-readable --json for every command.

sqldoc connects to your database, reads its schema (and, for a couple of commands, aggregate statistics — never row data), optionally uses an LLM to write plain-English descriptions, and produces reports you can open in any browser or feed to another tool. Documentation, PII scanning, schema intelligence, and AI insights run on all six engines via a common adapter layer; health and quality run on SQL Server, PostgreSQL, and MySQL. Pick the engine with --dialect or let it auto-detect from the connection string. SQLite uses the Python stdlib and SQL Server ships in the core install; the Postgres/MySQL/Snowflake drivers are optional installs (pip install sqldoc[postgres] / sqldoc[mysql] / sqldoc[snowflake] / sqldoc[all]).

Quick start

pip install sqldoc

# Document a database (schema-only, nothing leaves your machine):
sqldoc doc --server localhost --database AdventureWorks2022 \
    --username sa --password '***' --no-ai --output docs.html

# Scan for PII / regulated columns:
sqldoc scan --server localhost --database AdventureWorks2022 \
    --username sa --password '***' --output pii-report.html

Extraction needs the Microsoft ODBC Driver 18 for SQL Server on the host (download); it is a system package, not a pip dependency. AI descriptions are opt-in and, by default, run locally against Ollama — see the privacy section below.

The seven commands

Command	What it does	Reads
sqldoc doc	Schema documentation with AI descriptions, ER diagram, search. HTML / Markdown / PDF / JSON.	schema metadata
sqldoc scan	PII / compliance scan — ~21 categories mapped to HIPAA / GDPR / PCI-DSS, risk dashboard, SARIF, CI gate.	schema metadata
sqldoc health	DMV analysis — slow queries, dead tables, missing-index suggestions, index fragmentation.	server/DB statistics
sqldoc quality	Data-quality profiling — null rates, cardinality/distribution, duplicate detection.	data (aggregate only)
sqldoc intel	Schema intelligence — naming conventions, orphaned FKs, impact analysis, migration generation.	schema metadata
sqldoc insights	AI insights — natural-language-to-SQL, anomaly detection, business glossary, relationship inference.	schema metadata
sqldoc comply	Compliance reports — per-regulation HIPAA/GDPR/PCI-DSS scope + controls, data lineage, access audit.	schema + catalog metadata

Every command writes a self-contained dark-themed HTML report and accepts --json (or, for doc, --format json) for machine-readable output. sqldoc with options but no subcommand runs doc for backward compatibility.

🔒 Privacy guarantee

sqldoc runs on-premise by default and never reads your table row data for documentation, scanning, or schema analysis.

• Local by default. AI processing (in doc and insights) runs against an Ollama instance on your own machine unless you explicitly pass --mode cloud. Nothing leaves your network in local mode.
• Row data is never read for metadata work. doc, scan (without --sample), intel, insights, and comply query only sys.* catalog views and DMVs — no SELECT against your tables.
• The commands that do touch data say so and stay local. quality runs aggregate-only queries (COUNT / DISTINCT / MIN / MAX / GROUP BY) and prints a confirmation prompt; scan --sample reads ≤5 values per flagged column purely to score confidence and never stores them. Neither sends data off-network.
• Cloud is opt-in and explicit. --mode cloud prints a warning and requires confirmation before any network call, and sends only schema metadata (names, types, keys) — never row data. --include-definitions additionally sends view/procedure/trigger SQL bodies, with its own widened warning.

The worst-case disclosure in cloud mode is a column name like Employee.Salary — never a salary.

Installation

From PyPI:

pip install sqldoc              # SQL Server / Azure SQL (pyodbc) + SQLite (stdlib)
pip install sqldoc[postgres]    # + PostgreSQL (psycopg2)
pip install sqldoc[mysql]       # + MySQL (mysql-connector-python)
pip install sqldoc[snowflake]   # + Snowflake (snowflake-connector-python)
pip install sqldoc[all]         # + all optional drivers

The dialect is auto-detected from the connection string (postgresql://, mysql://, snowflake://, *.snowflakecomputing.com, *.db/*.sqlite, *.database.windows.net) or set explicitly with --dialect {sqlserver,azuresql,postgres,mysql,sqlite,snowflake}. doc, scan, intel, and insights run on all six engines; health and quality run on SQL Server, PostgreSQL, and MySQL; the comply access audit is SQL Server / Azure SQL only. (Snowflake is currently mock-tested, not yet validated against a live account.)

From source (editable/development install):

git clone https://github.com/htamber1/sqldoc.git
cd sqldoc
python -m venv venv
venv/Scripts/activate            # Windows;  source venv/bin/activate on macOS/Linux
pip install -e .[test]           # includes the pytest extras

Requirements: Python 3.10+, the ODBC Driver 18 for SQL Server, and — for AI descriptions — either a running Ollama (local, default llama3.1:8b) or an Anthropic API key (cloud). For cloud mode, put your key in a .env file:

ANTHROPIC_API_KEY=sk-ant-...

Commands in depth

sqldoc doc — documentation

Extracts tables, columns (incl. computed columns and constraints — PK/FK with referential actions, CHECK / UNIQUE / DEFAULT), indexes, triggers, views and procedures (with SQL definitions), enriches them with AI descriptions, and renders a single self-contained report.

# Local AI (Ollama), HTML with ER diagram + search:
sqldoc doc --server localhost --database AdventureWorks2022 --username sa --password '***'

# Markdown / PDF / JSON — inferred from the output extension, or forced with --format:
sqldoc doc ... --output docs.md
sqldoc doc ... --output docs.pdf
sqldoc doc ... --format json --output schema.json

The HTML report is an IDE-like reading experience: a collapsible sidebar navigation tree, an interactive ER diagram (schema-banded, FK arrows), real-time search + type filter, per-table Constraints sections, one-click Copy SQL on every definition, and color-coded row counts. --include-definitions sends view/proc/trigger bodies to the AI for richer descriptions (opt-in; widens the data boundary).

Schema change detection — every run snapshots the schema structure and the next run prints a git-style diff (new/dropped tables & columns, type/key/constraint changes). --snapshot / --no-snapshot.

sqldoc scan — PII / compliance

Flags columns likely to hold personal or regulated data by name + type, maps each to HIPAA / GDPR / PCI-DSS, rates HIGH / MEDIUM / LOW with a numeric confidence score, and writes a compliance dashboard (with CSV export).

sqldoc scan --server localhost --database AdventureWorks2022 --username sa --password '***'

• --confidence-threshold 0.0-1.0 drops weak (name-only / type-mismatch) matches.
• --sample reads ≤5 values per column and asks the AI to confirm (values never stored; prompts first).
• pii_allowlist: in .sqldoc.yml suppresses known-safe columns (schema.table.column, bare column, or a glob like dbo.*.Password).
• pii_patterns: defines org-specific categories.
• PII drift (--baseline), SARIF 2.1.0 (--sarif) for GitHub Advanced Security / Azure DevOps, JSON (--json), and a CI gate (--fail-on high|new-high).

sqldoc health — DMV analysis

Reads SQL Server DMVs (statistics only) for the slowest cached queries, dead tables (rows + writes but no reads), optimizer missing-index suggestions with a generated CREATE INDEX, and fragmented indexes with a REBUILD/REORGANIZE call. Each check is isolated: a missing VIEW SERVER STATE degrades that section, not the whole run. --top, --min-fragmentation, --min-pages.

sqldoc quality — data-quality profiling

Profiles the data in aggregate only: per-column null rate (with a high-null flag), distinct count/cardinality, min/max, blank counts, most-frequent values (--top-values), and full-row duplicate detection (--no-duplicates to skip). Prints a local-only notice and confirms before running (--yes to skip).

sqldoc intel — schema intelligence

Naming-convention analysis (dominant style + outliers, PK naming), orphaned-FK detection (implied-but-unenforced relationships), impact analysis ("what breaks if you drop this table"), and migration-script generation from a baseline snapshot (--baseline snapshot.json, --migration-out migration.sql).

sqldoc insights — AI insights

sqldoc insights --server localhost --database AdventureWorks2022 --username sa \
    --password '***' --ask "which customers placed the most orders last month?"

• Natural-language-to-SQL — --ask "question" (repeatable) returns a schema-grounded T-SQL query.
• Anomaly detection (heuristic, always on) — tables with no primary key, generic column names, missing audit columns, and name/type mismatches (a *Date stored as varchar, etc.).
• Business glossary — an AI-inferred term + definition per table, rendered as a searchable glossary (--no-glossary to skip).
• Relationship inference — likely foreign keys with a confidence score and a ready-to-run ALTER TABLE … ADD CONSTRAINT.

--no-ai still runs the heuristic anomaly + relationship analysis.

sqldoc comply — compliance expansion

• Per-regulation reports — the scan findings grouped by HIPAA / GDPR / PCI-DSS, each showing the regulated columns and the controls that regime typically requires.
• Data lineage — flows through view/procedure SQL (a view reads its source tables; a proc's INSERT … SELECT is a directional write).
• Access audit — object-level grants from sys.database_permissions cross-referenced with the PII findings ("who can read regulated columns"); --no-access-audit if the account lacks VIEW DEFINITION.

Config file

Rather than repeating flags, drop a .sqldoc.yml in the working directory — every key maps to the CLI option of the same name (an explicit flag wins over config, which wins over defaults). Copy .sqldoc.example.yml to start.

server: localhost
database: AdventureWorks2022
username: sa
mode: local
# password: better supplied via --password or .env than committed to disk
pii_allowlist:
  - dbo.Config.ContactEmail       # a support inbox, not personal PII

.sqldoc.yml is gitignored because it can contain a password. Keep secrets out of it (use --password or .env) if you plan to share it.

How does it compare?

sqldoc overlaps with commercial documentation and data-catalog tools but takes a free, CLI-first, multi-database, privacy-first angle. The table below is based on publicly documented capabilities — verify current features against each vendor.

Capability	sqldoc	Redgate SQL Doc	Dataedo
Price	Free / open source (MIT)	Paid (per-user)	Paid (per-user / repo)
Interface	CLI — scriptable, CI-friendly	Desktop GUI + CLI	Desktop app + web repo
Databases	SQL Server, PostgreSQL, MySQL, SQLite, Snowflake, Azure SQL	SQL Server	20+ (SQL Server, Oracle, PostgreSQL, MySQL, …)
Self-contained HTML docs	✓	✓ (HTML / CHM / Word / Markdown)	✓ (web catalog)
AI-written descriptions	✓ (local Ollama or cloud)	✗	Partial (AI assist)
Runs fully offline / on-prem	✓ (local mode, no data egress)	✓	✓ (on-prem repo)
PII / sensitive-data detection	✓ (+ HIPAA/GDPR/PCI mapping)	✗	✓ (classification)
Compliance reports (HIPAA/GDPR/PCI-DSS)	✓	✗	Partial
DMV health (slow queries, missing indexes)	✓	✗	✗
Data-quality profiling (nulls, dupes)	✓	✗	✓
Business glossary	✓ (auto-generated)	✗	✓ (curated catalog)
Data lineage	✓ (view/proc parsing)	✗	✓ (advanced, cross-object)
Schema change detection / diff	✓	Partial	✓
Machine-readable JSON output	✓ (every command)	Partial	✓ (API)
Natural-language-to-SQL	✓	✗	✗

Where each shines. Redgate SQL Doc is a polished static-documentation generator for SQL Server (Word/CHM output, source-control integration). Dataedo is a mature, multi-database data catalog with rich curated lineage and glossary and a team repository. sqldoc is free and scriptable, focuses on SQL Server, adds AI descriptions and a privacy-first local mode, and bundles PII/compliance, DMV health, and data-quality analysis that the documentation tools don't cover.

How it works

A linear pipeline, one module per stage, orchestrated by cli.py:

• extractor.py — the only DB-facing code for the metadata path; queries sys.* catalog views and builds the shared dataclasses (Table / Column / Index / View / StoredProcedure / constraints).
• ai.py — fills descriptions via Ollama (local) or the Anthropic SDK (cloud), concurrently, with retry + a structural description cache.
• Renderers — renderer.py (HTML), markdown_renderer.py, pdf_renderer.py, json_renderer.py, plus a dedicated *_renderer.py per analysis command.
• Analysis modules — pii.py, health.py, quality.py, intel.py, insights.py, comply.py, each with a build_*_json for machine-readable output.

Run the test suite (mocked — no live SQL Server or Ollama required) with pytest -q.

License

Released under the MIT License — © 2026 Harsh Tamboli. You are free to use, modify, and redistribute sqldoc, including commercially, provided the license and copyright notice are preserved.