bigquery-data-science

Production Agent Skills, zero-dependency CLI & stdio MCP server for BigQuery ML and BigQuery AI — distilled directly from 104 live-executed notebooks and 87 SQL scripts in Google Cloud.

104 live-verified notebooks · 87 SQL scripts · 3 Agent Skills · 137 <25 KB syntax cards · zero-dependency CLI & stdio MCP

What you just found

This repository pairs two complementary layers that stay synchronized through automated tooling (bigquery-data-science sweep):

1. Ground-Truth Notebooks, SQL & Reference Guides (bigquery-ml/ & bigquery-ai/): Every model type, ML.* function, AI.* function, composed workflow, and orchestration pipeline is validated end-to-end in a live-executed Jupyter notebook (.ipynb with stored execution outputs and one-click Colab / Vertex AI Workbench / BigQuery Studio headers) paired with a pure SQL companion script (.sql) and 13 exhaustive domain reference guides.
2. 4-Lens Agent Skills, CLI & Stdio MCP Server (skills/ & src/bigquery_data_science/): Instead of dumping 24 MB of raw notebooks or 200+ KB monolithic reference files into an AI coding agent's context window, bigquery-data-science compiles the ground-truth notebooks and references into 4 progressive lenses — routing agents from an ~8 KB SKILL.md decision tree to a surgical <25 KB syntax card (syntax/*.md) or a clean markdown walkthrough (narrative/*.md) only when needed.

Three ways to start

Journey 1 — Equip your AI coding agent (uvx, uv, pip, pipx, npx skills, or MCP)

The core bigquery-data-science package has zero third-party runtime dependencies (< 1.5 MB wheel) and installs the 3 skills (bigquery-data-science, bigquery-ml, bigquery-ai) into .agents/skills/ and .claude/skills/:

# Option A — Zero-install ephemeral CLI via uvx (recommended for instant setup):
uvx bigquery-data-science install                     # installs into ./.agents/skills & ./.claude/skills
uvx bigquery-data-science install --mcp               # also registers stdio MCP server in ./.mcp.json
uvx bigquery-data-science install --global            # installs into ~/.agents/skills & ~/.claude/skills
uvx bigquery-data-science install --target agents     # only .agents/skills (Antigravity, Cursor, Codex, Windsurf)
uvx bigquery-data-science install --target claude     # only .claude/skills (Claude Code)

# Option B — Persistent CLI install via uv tool, pipx, or pip:
uv tool install bigquery-data-science && bigquery-data-science install
pipx install bigquery-data-science && bigquery-data-science install
pip install bigquery-data-science && bigquery-data-science install

# Option C — Add to an existing Python project (uv or pip):
uv add bigquery-data-science && uv run bigquery-data-science install
pip install bigquery-data-science && python3 -m bigquery_data_science.cli install

# Option D — skills.sh CLI (installs all 3 skills or an individual skill from GitHub):
npx skills add statmike/bigquery-data-science
npx skills add statmike/bigquery-data-science --skill bigquery-ml

# Option E — Claude Code plugin marketplace & stdio MCP server:
# In Claude Code: /plugin marketplace add statmike/bigquery-data-science
#                 /plugin install bigquery-data-science
claude mcp add bigquery-data-science -- uvx bigquery-data-science mcp
# Or with a local pip/uv install:
claude mcp add bigquery-data-science -- python3 -m bigquery_data_science.cli mcp

Journey 2 — Triage tasks & look up exact BigQuery syntax from the terminal

Run any catalog command directly with uvx bigquery-data-science ... (zero install) or bigquery-data-science ... (after uv tool install or pip install):

# 1. List bundled skills and lens counts:
uvx bigquery-data-science list

# 2. Triage an ambiguous BigQuery ML vs BigQuery AI task (10 empirical head-to-head comparisons):
uvx bigquery-data-science triage "forecast daily retail sales with holiday promotions"

# 3. Fetch the surgical syntax card (< 25 KB) for any model or function:
uvx bigquery-data-science lookup ARIMA_PLUS
uvx bigquery-data-science lookup AI.FORECAST
uvx bigquery-data-science lookup ML.ENTITY_FEATURES_AT_TIME

# 4. Search across all 137 syntax cards, 14 reference guides, and 104 narratives:
uvx bigquery-data-science search "point-in-time feature leakage"

# 5. Statically audit a BigQuery SQL query or .sql file for prerequisites & gotchas:
uvx bigquery-data-science review-sql "SELECT AI.GENERATE('Summarize', endpoint => 'gemini-3.5-flash')"

# 6. Add --json to any catalog command (list, lookup, search, triage, review-sql) for automation:
uvx bigquery-data-science lookup AI.AGG --json

Journey 3 — Run the 104 ground-truth notebooks & SQL scripts (uv or pip)

Every notebook in bigquery-ml/ and bigquery-ai/ includes committed outputs for immediate reading on GitHub and one-click launch links for Colab, Colab Enterprise, Vertex AI Workbench, and BigQuery Studio.

To run notebooks locally, choose the dependency extra that matches the folders you want to execute:

git clone https://github.com/statmike/bigquery-data-science.git
cd bigquery-data-science

# With uv (recommended — uses the locked uv.lock environment):
uv sync --extra all
uv run bigquery-data-science kernel --user

# Or with standard Python venv + pip:
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[all]"
bigquery-data-science kernel --user

Front Door 1 — Agent Skills & 4-Lens Progressive Disclosure (skills/)

The 3 Bundled Skills

How the 4 Progressive Lenses Prevent Context Bloat

Front Door 2 — Ground-Truth Notebooks, SQL & Reference Hubs (bigquery-ml/ & bigquery-ai/)

Decoding BigQuery's ML.* vs AI.* SQL Namespaces

BigQuery exposes machine learning and generative AI across two SQL prefixes (ML.* and AI.*) that reflect when a function was introduced and whether it requires a CREATE MODEL DDL object. Full matrix: skills/bigquery-data-science/reference/namespace-matrix.md.

Navigate the Notebook & Reference Hubs

Instead of scrolling through 104 individual rows here, jump directly into the domain hubs and category indexes below:

Contributing, Pre-Commit & CI/CD

Contributions are welcome — see CONTRIBUTING.md, CODE_OF_CONDUCT.md, and SECURITY.md.

• GitHub Actions CI & Release Workflows: .github/workflows/ci.yml runs ruff, mypy, pytest, bigquery-data-science sweep, sync_headers.py --check, and ./scripts/check_install.sh across Python 3.11, 3.12, and 3.13 on every push and pull request. .github/workflows/release.yml publishes release candidates (v*rc*) to TestPyPI and production tags (v*) to PyPI via OIDC Trusted Publishing.
• Local Pre-Commit Hooks (.pre-commit-config.yaml): Install with uv run pre-commit install to run ruff, mypy, bigquery-data-science sweep, and sync_headers.py --check automatically before each commit.

Run the complete local verification and cold-install gate before submitting a pull request:

uv run ruff check src tests && \
  uv run mypy src && \
  uv run pytest -q && \
  uv run bigquery-data-science sweep && \
  python3 architectures/headers/sync_headers.py --check && \
  ./scripts/check_install.sh

Versioning & Release Cadence

bigquery-data-science adheres to Semantic Versioning (Major.Minor.Patch):

• Major (X.0.0) — Skills architecture, names & structural contract shifts: Changes in skills hierarchy/naming (skills/), packaging overhauls, or breaking changes to CLI subcommands, MCP tool schemas, or library exports.
• Minor (X.Y.0) — Multi-area monthly sweeps & capability rollouts: Scheduled monthly sweeps across BigQuery release notes — introducing new foundation models, options, notebooks, SQL workflows, and synchronized 4-lens skill updates.
• Patch (X.Y.Z) — Targeted single-area updates & fixes: Isolated bug fixes, typo corrections, single-notebook adjustments, or targeted hotfixes.

See CONTRIBUTING.md for the full release process and TestPyPI/PyPI tag flow.

License

Apache-2.0 — see LICENSE.

This is not an officially supported Google product. It is built on Google products and is neither endorsed nor supported by Google.