doc3gpp

Last reviewed: 2026-08-13

Provide 3GPP TDoc information with full text search capability — a Python CLI/library, a local web UI, and an MCP server for AI clients.

Description

doc3gpp scrapes 3GPP meeting calendars, work items (WIs), TDocs, and specifications (TS/TR) from 3gpp.org and persists them to a relational database for programmatic access. It ships in three forms:

• a Python library (SDK) for embedding the data in your own code,
• a Typer-based CLI (doc3gpp) for scripting and ad-hoc queries, and
• a local web server with a built-in browser UI (FastAPI + HTMX + Jinja2) plus a Model Context Protocol endpoint (/mcp) that exposes the same data to AI clients with byte-for-byte JSON parity against the HTTP ?format=json routes.

SQLite is the sole storage backend.

Table of Contents

• Features
• Installation
• Quick Start
• CLI Usage
• Database Configuration
• Configuration File (TOML)
• Architecture
• Testing
• Roadmap
• Documentation
• Contributing
• License
• Acknowledgments
• Support

Features

• Meeting / TDoc / WI / Spec sync — fetch the 3GPP Meetings, TDocs, Work Items List, and 3GPP specifications (TS/TR) with versions. Spec sync fans out across per-spec detail pages in a thread pool and caches the result for spec list / spec show.
• RAN5 testcase sync — fetch the latest TTCN CR Agreement Status snapshot (testcase sync) and query it via testcase list / testcase show (per-path ttcn_status / gcf_ptcrb).
• TDoc extraction — Download and parse TDoc CRs into structured records, and store the Markdown of CR annexes and WID, LS, and discussion TDocs for raw retrieval and search.
• Unified search and indexing — TDoc and spec-document search support FTS5, semantic, hybrid, and filter-only modes; companion index commands maintain the FTS5 and vector components.
• Hybrid semantic search (FTS5 + embeddings) — vector KNN + FTS5 keyword search, merged via reciprocal-rank fusion.
• Web server + MCP — a single-port HTTP server (FastAPI + HTMX + Jinja2) for browsing and searching 3GPP data in a browser, plus a Streamable-HTTP Model Context Protocol endpoint (/mcp) exposing the same data to AI clients with byte-for-byte JSON parity with the HTTP ?format=json routes. Background jobs (sync, parse, unified search-index maintenance, cache purge) run on a shared asyncio worker with live SSE progress.
• SQLite storage backend — via SQLAlchemy 2.0.

Installation

SDK (library)

pip install doc3gpp

Use the SDK to access 3GPP data programmatically:

from doc3gpp.services.meetings_service import MeetingService
from doc3gpp.storage.repositories.meeting_sql import SQLAlchemyMeetingRepository

service = MeetingService(SQLAlchemyMeetingRepository())
meetings = service.list_recent(limit=10)

CLI (command-line tool)

pip install "doc3gpp[cli]"
# or, for an isolated install:
pipx install "doc3gpp[cli]"

The [cli] extra adds the doc3gpp command (Typer-based subcommands).

Development

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
cp .env.example .env
doc3gpp db init
doc3gpp db check

The [dev] extra includes [cli], pytest, pytest-cov, and ruff.

Optional extras

pip install "doc3gpp[extract]"    # TDoc CR extraction (python-docx)
pip install "doc3gpp[search]"     # FTS5 + BM25 full-text search
pip install "doc3gpp[semantic]"   # Hybrid FTS5 + embedding vector search (sqlite-vec; remote embeddings API)
pip install "doc3gpp[web]"        # Web server + MCP (FastAPI, uvicorn, Jinja2 + HTMX, markdown-it-py)
pip install "doc3gpp[all]"        # every runtime extra: CLI, extraction, search, semantic, web

Quick Start

SDK

from doc3gpp.services.meetings_service import MeetingService
from doc3gpp.services.tdoc_service import TDocService
from doc3gpp.services.wi_service import WiService
from doc3gpp.storage.repositories.meeting_sql import SQLAlchemyMeetingRepository
from doc3gpp.storage.repositories.tdoc_sql import SQLAlchemyTDocRepository
from doc3gpp.storage.repositories.wi_sql import SQLAlchemyWiRepository

meetings = MeetingService(SQLAlchemyMeetingRepository())
tdocs = TDocService(SQLAlchemyTDocRepository())
wis = WiService(SQLAlchemyWiRepository())

recent = meetings.list_recent(limit=5)
for m in recent:
    print(m.meeting_id, m.name, m.end_date)

CLI

doc3gpp db init                            # create schema + seed tsgs table
doc3gpp meeting sync --tsg r5              # scrape DynaReport, validate --tsg
doc3gpp meeting list --limit 5
doc3gpp meeting list --tdoc R5-260013       # find the meeting containing a TDoc
doc3gpp tdoc sync --meeting-id 85434       # requires a stored meeting row
doc3gpp tdoc sync                          # sync every tracked meeting_id in tdocs
doc3gpp tdoc list --tdoc 'R5%'
doc3gpp tdoc parse --meeting-id 85434      # extract CR cover pages; prompts before batch
doc3gpp tdoc parse --tdoc 'R5s26%' --yes   # pattern match, skip confirmation
doc3gpp wi sync --tsg r5                   # scrape WI DynaReport for R5
doc3gpp wi list --release "Rel-19" --limit 50
doc3gpp spec sync --tsg r5                 # scrape spec list + parallel detail pages
doc3gpp spec sync --spec-id 36.579-5       # sync a single stored spec
doc3gpp spec list --type TS --limit 20
doc3gpp spec show 36.579-5                 # header + version rows
doc3gpp spec doc parse --spec 38.331         # fetch, convert, chunk, and index a spec zip
doc3gpp testcase sync                      # latest TTCN status History zip → testcases
doc3gpp testcase list --group 5G --limit 20
doc3gpp testcase show --testcase TC_1      # header + per-path status rows

CLI Usage

The CLI ships twelve sub-apps. The most common entry points are meeting sync (DynaReport calendar), tdoc sync (TDoc-list XLSX + auxiliary file scan), tdoc parse (extract CR covers and ZIP document content), tdoc show --format raw (render converted Markdown for supported .docx members), spec sync (3GPP specification records with versions), spec doc parse (specification document conversion and chunking), and tdoc search / spec doc search (unified FTS5, semantic, hybrid, and filter-only search).

db — database lifecycle

doc3gpp db init                # create schema + seed tsgs table
doc3gpp db check               # verify connectivity
doc3gpp db reset --yes         # destructive: wipe + recreate SQLite schema

tsg — 3GPP TSG reference

doc3gpp tsg list               # show the canonical 3GPP TSG list
doc3gpp tsg show --tsg r5      # show a single TSG record
doc3gpp tsg seed               # re-seed the reference table
doc3gpp tsg schema --format json  # describe every tsgs column (meaning, format, values)

meeting — 3GPP meeting calendar

doc3gpp meeting sync --tsg r5              # scrape DynaReport; --tsg validated against tsgs
doc3gpp meeting list --limit 20
doc3gpp meeting list --tdoc R5-260013      # find the meeting whose start_doc/end_doc brackets a TDoc
doc3gpp meeting schema --format json       # describe every meetings column

tdoc — list, parse, show

# sync — every tracked meeting_id or one specific meeting
doc3gpp tdoc sync                                # sync every distinct meeting_id in tdocs
doc3gpp tdoc sync --meeting-id 85434
doc3gpp tdoc sync --meeting "R5--TTCN Workshop#74"

# list — 18 filter flags combine freely
doc3gpp tdoc list --limit 10
doc3gpp tdoc list --tdoc 'R5%'                   # LIKE pattern on tdoc_id
doc3gpp tdoc list --meeting-id 85434 --cr-cat F
doc3gpp tdoc list --tdoc 'R5%' --meeting "%RAN3%"
doc3gpp tdoc list --title '!%Sidelink%'          # NOT LIKE
doc3gpp tdoc list --parsed true                  # only TDocs with a complete extraction (parsed column)

# parse (DB mode) — every flag is a filter
doc3gpp tdoc parse --meeting-id 85434            # CR, WID, LS, and discussion TDocs (default); prompts before batch (pending only)
doc3gpp tdoc parse --tdoc 'R5s26%' --yes         # LIKE pattern; non-interactive
doc3gpp tdoc parse --meeting-id 85434 --meeting '%RAN5%' --cr-cat F
doc3gpp tdoc parse --meeting-id 85434 --release 'Rel-19' --cr-num not-null
doc3gpp tdoc parse --meeting-id 85434 --force    # re-extract everything (includes already-parsed)

# parse (direct mode) — bypasses DB filters
doc3gpp tdoc parse --from-path ~/Downloads/R5s260009.docx                # local .docx → stdout
doc3gpp tdoc parse --from-url https://www.3gpp.org/ftp/.../R5s260009.zip # 3GPP URL → cache + DB
doc3gpp tdoc parse --from-url https://example.com/some.zip --format json -o /tmp/out.json  # non-3GPP URL → in-memory only
doc3gpp tdoc parse --from-path ./tdocs --output ./parsed --recursive --format json         # local batch
doc3gpp tdoc parse --from-url https://www.3gpp.org/ftp/.../Docs/ --recursive --output ./parsed  # online batch
# 3GPP URLs also accept bare www.3gpp.org/ftp/...; HTTPS is required,
# and .docx/.zip basenames must contain a TDoc id (including RP-... plenary ids).

# show — --tdoc and --ftp-url are mutually exclusive
doc3gpp tdoc show --tdoc R5s260009 --format json -o r5s260009.json
doc3gpp tdoc show --tdoc R5s260009 --format raw  -o r5s260009.md    # all converted ZIP documents
doc3gpp tdoc show --tdoc R5s260009 --format raw --source-file 'Annex/Support.docx' -o support.md
doc3gpp tdoc show --ftp-url tsg_ran/WG5/.../R5s260009.zip            # URL-keyed lookup
doc3gpp tdoc show --ftp-url https://www.3gpp.org/ftp/.../R5s260009.zip --format raw

# schema — describe every column of the ten tdoc tables (99 rows, no filters)
doc3gpp tdoc schema --format json

TDoc ZIP parsing converts .docx members only; .doc, PDF, spreadsheets, and nested ZIPs are not converted, so a TDoc whose main document is only a .doc file fails. For a CR TDoc, the document that contains the CR cover is the main document: it owns the cover and optional TTCN/body-change sidecars, and every other .docx in the archive is a CR annex stored as content. WID, LS, and discussion TDocs store their main document and every annex. A failed annex is skipped with a warning that names it, is listed in skipped_sources, and leaves the extraction incomplete until a later normal parse converts it. Use --source-file with --format raw to select one full member path; without it, multiple bodies are emitted as named blocks. tdoc_files remains the metadata for auxiliary files discovered in meeting FTP folders—it is not the inventory of files inside a ZIP.

Content is stored in SQL under the single tdoc_extracts parent for each ZIP URL, in one table per store: tdoc_cr_annexes for CR annexes, and tdoc_wid_contents, tdoc_ls_contents, and tdoc_discussion_contents for family documents. The CR main Markdown is never stored in SQL; it lives only in the derived Markdown archive. That archive and its manifest are derived cache artifacts: purging or evicting them does not remove stored content or prevent an FTS5 rebuild, but raw CR main Markdown is unavailable until the TDoc is re-downloaded. Legacy plain-text, gzip, and single-member Markdown cache artifacts remain readable as incomplete version-1 content; parse discovery treats those extractions as pending so the ordinary filtered parse workflow can upgrade them. A URL-anchored raw read is a snapshot and does not implicitly download or extract; the TDoc-ID raw path retains its existing extraction-on-read behavior.

wi — Work items

doc3gpp wi sync --tsg r5                       # scrape the WI DynaReport page for R5
doc3gpp wi list --limit 10                     # default fields: wi_id, acronym, release, name
doc3gpp wi list --tsg R5 --release "Rel-19" --limit 100
doc3gpp wi schema --format json                # describe every wis column

spec — 3GPP specifications (TS / TR)

# sync — fetch list page + parallel detail pages (TSGs / WI links / ETSI PDF / CR list)
doc3gpp spec sync --tsg r5                          # 24h skip rule via sync.spec_sync_interval
doc3gpp spec sync --tsg r5 --force                  # bypass the skip rule
doc3gpp spec sync --spec-id 36.579-5                # sync a single stored spec (no list page)
doc3gpp spec sync --spec-id 36.579-5 --force        # bypass the skip rule for one spec
doc3gpp spec sync --tsg r5 --per-version-details    # always re-fetch ETSI PDF + CR list per version (default OFF)

# list — filters combine freely (rich-filter grammar: %, !pattern, null, not-null;
# --parsed accepts exactly true or false)
doc3gpp spec list --limit 20
doc3gpp spec list --tsg R5 --type TS --status "Under change control"
doc3gpp spec list --spec-id '36.579%' --title '%conformance%'
doc3gpp spec list --tsg R5 --format json -o r5_specs.json
doc3gpp spec list --parsed true --format json
doc3gpp spec list --parsed false --limit 20

# show — dotted spec id; renders header + version rows
doc3gpp spec show 36.579-5
doc3gpp spec show 36.579-5 --format json -o 36_579-5.json

# schema — describe every column of specs + spec_versions (20 rows, no filters)
doc3gpp spec schema --format json

spec sync honours sync.spec_sync_interval (default 24h) on a per-spec basis: each per-worker _sync_one_spec short-circuits specs whose own specs.last_synced_at is within the interval, and re-syncs the rest; --force bypasses the check. Each spec's last_synced_at is stamped on a successful re-sync so a spec list / spec show always reads from cache. Per-version follow-ups (ETSI PDF + CR list) are skipped by default; pass --per-version-details to fetch them. The default preserves any previously-cached pdf_url / crs values on existing rows.

The default spec list fields end with rapporteurs, parsed. Parsed status is derived from non-null spec_doc_sources.parsed_at values in the separate specdata database; it is not a column in the main specs or spec_versions tables and is not part of spec schema. List JSON returns a comma-separated, numeric-newest-first parsed-version string or native null; table and Markdown output display -. The --parsed true|false filter is applied before pagination. spec show --format json returns native boolean parsed values on every version row.

spec doc — specification document corpus

# use newest version with a published .zip, fetch-if-missing, convert every .docx, chunk, and auto-index
doc3gpp spec doc parse --spec 38.331 --release Rel-18 --version 18.5.0
doc3gpp spec doc parse --spec 38.331 --force  # re-download and re-parse

# inspect the stored TOC and search chunk text/combined metadata
doc3gpp spec doc toc show --spec 38.331 --version 18.5.0 --format json
doc3gpp spec doc search --text "handover" --spec 38.331 --sections "%handover%" --tables "%UE%" --limit 20
doc3gpp spec doc search --semantic "handover" --spec 38.331 --limit 20
doc3gpp spec doc index --rebuild
doc3gpp spec doc schema --format json

Background parse_spec_docs jobs retain the selected spec's failure reason in the job result (failure_details) and log. Each job targets one spec; when both release and version are supplied, their release numbers must align (for example, Rel-18 with 18.5.0).

Spec-document rows live in a separate sibling database, normally <main-stem>_specdata.db, and are cached below the dedicated ~/.cache/doc3gpp/specs root by default. The TDoc extraction cache remains ~/.cache/doc3gpp/tdocs. Chunk sections and tables values are newline-delimited combined identifier/title entries; TOC entries retain their section_no and title fields. Section identifiers may be alphanumeric, such as 7.2A.3 or 7.2A.3A. Version selection is numeric rather than lexical; parsed (spec_id, version) rows are immutable and skip on later parses unless --force is supplied, which re-downloads and re-parses. The parse command fetches the ZIP when it is missing and reports ok, skipped, and failed buckets per requested spec. See docs/cli.md for all filters and defaults.

The pre-deployment specdata corpus is repaired once from its cached ZIPs using an internal maintenance helper. This is not a runtime migration and there is no public repair command; deployments should start from the fresh schema and parse ledger.

testcase — RAN5 conformance testcases

# sync — latest TTCN CR Agreement Status History zip (file-identity skip)
doc3gpp testcase sync
doc3gpp testcase sync --force            # re-download + re-parse even when recorded

# list — 10 filter flags combine freely (rich-filter grammar: %, !pattern, null, not-null)
doc3gpp testcase list --limit 20
doc3gpp testcase list --group 5G --spec '38.523-1'
doc3gpp testcase list --status Approved --format json -o testcases.json

# show — one testcase id; renders header + per-path status rows
doc3gpp testcase show --testcase TC_1
doc3gpp testcase show --testcase TC_1 --format json -o tc_1.json

# schema — describe every column of the three testcase tables (21 rows, no filters)
doc3gpp testcase schema --format json

testcase list --status / --gcf-status match ttcn_status / gcf_ptcrb on ANY path (EXISTS); list JSON emits flat per- (testcase_id, group) objects with a nested statuses list of {path, gcf_ptcrb, ttcn_status} objects (no testcase envelope, no group in status rows; None stays null in JSON). Default columns testcase_id,title,spec,group,release,statuses (TOML [output.fields] testcase). testcase show JSON emits the same flat per-(testcase_id, group) shape with all 8 header fields inline plus the nested statuses list.

tdoc search — unified TDoc search

The command accepts independent --text and --semantic inputs. Text-only search uses FTS5, semantic-only search uses the vector index, both inputs use hybrid RRF search, and neither input runs filter-only source SQL.

# FTS5 text search
doc3gpp tdoc search --text "NB-IoT scheduling" --tsg RAN1 --limit 10
doc3gpp tdoc search --text "R5-1234567" --format json
# FTS5 can find text present only in a CR annex or family document
doc3gpp tdoc search --text "annex-specific handover parameter" --format json

# vector search
doc3gpp tdoc search --semantic "power saving for NB-IoT UEs" --limit 10

# hybrid search; [semantic_search].fts5_weight controls the blend
doc3gpp tdoc search --text "NB-IoT" --semantic "power saving" --limit 10

# filter-only source SQL
doc3gpp tdoc search --tsg RAN1 --spec 38.300 --since 2026-01-01

# status and maintenance for both indexes
doc3gpp tdoc index
doc3gpp tdoc index --rebuild --resume --batch 1000
doc3gpp tdoc index --rebuild-all

The auto-index and auto-embed hooks keep the indexes fresh after successful parses when enabled in doc3gpp.toml.

TDoc text syntax

tdoc search --text uses the FTS5 rich-text search pattern. Plain text is wrapped as a quoted expression (implicit AND between terms), while queries with FTS5 operators pass through unchanged:

doc3gpp tdoc search --text "scheduling AND (NR OR LTE)"
doc3gpp tdoc search --text "R5-*"                       # prefix match on TDoc ids
doc3gpp tdoc search --text "38.300*"                    # prefix match on spec ids
doc3gpp tdoc search --text '\"CSI report\" NEAR/5 feedback'

Search JSON contract

CLI JSON, HTTP JSON, and MCP search rows share the same flattened shape. The array is already in final rank order. search_mode is one of fts5, semantic, hybrid, or filter; score is respectively the FTS5 BM25 score, vector distance, RRF score, or JSON null for filter mode. previews is an object only for FTS5 rows and JSON null for the other modes. TDoc rows also include type, status, and best_chunk_id. Public rows do not expose per-source rank fields, nested hits, or separate distance/RRF fields. Every TDoc row also contains matched_documents: an array of {document_id, ftp_url, source_file, score, previews} source matches. FTS5 and hybrid results may include matches (hybrid preserves the FTS5 source list); semantic-only and filter-only rows always contain an empty array. Use each entry's exact ftp_url and full source_file for a source-specific content read; the top-level ftp_url retains its canonical-parent meaning. Stored content documents (CR annexes, and the main and annexes of WID, LS, and discussion TDocs) are FTS5-searchable and semantically chunked, so semantic-only search can match them. The CR main body is not full-text indexed; its cover, change, and TTCN fields are.

The request-level --fts5-weight, --fts5-query, and --sem-query flags were removed. Supply text with --text, semantic input with --semantic, and configure hybrid weighting once with [semantic_search].fts5_weight.

config — TOML config lifecycle

doc3gpp config init                              # bootstrap a TOML config file with full defaults
doc3gpp config path                              # which file is in effect (or "(no config file found)")
doc3gpp config show                              # fully-resolved Settings as JSON for diffing
doc3gpp config set sync.auto_sync true           # write one setting into the active TOML config
doc3gpp config set sync.auto_sync true --dry-run # preview the resulting TOML without writing

cache — Local extraction cache

doc3gpp cache status                  # file count, total bytes, limit, per-subdir breakdown
doc3gpp cache purge --yes             # delete cached markdown sidecars (default scope)
doc3gpp cache purge --scope zips --yes # only the 3GPP-served zip blobs
doc3gpp cache purge --scope all --yes  # both subtrees

Common output options

Every * list command accepts --format {table,json,markdown} and -o/--output PATH. meeting list, tdoc list, and tsg list additionally accept --fields to override the configured column set (wi list and spec list use their configured output.fields.wi / output.fields.spec lists):

doc3gpp tdoc list --format json -o tdocs.json
doc3gpp meeting list --format markdown -o meetings.md
doc3gpp tsg list --format json
doc3gpp wi list --format markdown

tdoc show accepts the same --format + -o/--output pair plus --format raw for the converted Markdown. Raw output emits one document unchanged, or several documents as named source blocks with the main document first. Direct-mode tdoc parse --from-path / --from-url and folder batches also accept --format raw with the same rules.

• --compact — strip output formatting. JSON drops indent and operator space (single line, separators=(",", ":")); Markdown drops CommonMark decorators (bold, italic, headings, bullets, GFM tables, code fences) and emits key: value lines with blank-line section separators. No-op for table and raw. Default: false; opt in globally with [output] compact = true in doc3gpp.toml.

Full command reference: docs/cli.md.

server — web server + MCP

The optional doc3gpp[web] extra installs a single-port HTTP server that serves both a browsable HTMX UI and a Model Context Protocol endpoint. It is enabled by default and binds only to loopback at port 13999; install an OS service if desired, then start it:

doc3gpp server install systemd --no-start     # or `launchd` on macOS
doc3gpp server start                          # opens http://127.0.0.1:13999/

• HTML UI — browse meetings, TDocs, TSGs, WIs, specs, spec-document TOCs, and unified FTS5, semantic, hybrid, and filter-only search results.
• JSON API — every read route accepts ?format=json, byte-for-byte identical to the MCP tools. Unified search is available at GET /tdocs/search and GET /spec-docs/search; index status is available at GET /tdocs/index and GET /spec-docs/index. Index maintenance jobs are enqueued with POST /jobs/tdocs/index and POST /jobs/spec-docs/index. Raw TDoc Markdown is served by GET /tdocs/{tdoc_id}/content and GET /tdocs/by-url/content?ftp_url=... (format=markdown by default, or format=json); these reads never download. Removed /search/sem routes are not current APIs.
• MCP — http://127.0.0.1:13999/mcp exposes the same reads and jobs. Search tools are search_tdoc and search_spec_docs; index tools are get_tdoc_index, index_tdocs, get_spec_doc_index, and index_spec_docs. Search rows are flattened and JSON-parity is preserved across CLI, HTTP, and MCP.
• Jobs — sync, parse, unified index maintenance, and cache purge run on a shared asyncio worker; watch live progress over SSE at /jobs/{id}/events.

Point an MCP client at the endpoint. For example, in Claude Desktop's claude_desktop_config.json (or any client that supports a Streamable-HTTP MCP server):

{
  "mcpServers": {
    "doc3gpp": {
      "type": "http",
      "url": "http://127.0.0.1:13999/mcp"
    }
  }
}

For clients that only speak the legacy SSE protocol, switch the server transport and point at the SSE endpoint instead:

doc3gpp config set mcp.transport sse

{
  "mcpServers": {
    "doc3gpp": {
      "type": "sse",
      "url": "http://127.0.0.1:13999/mcp/sse"
    }
  }
}

Manage the process with doc3gpp server start|stop|status|logs, doc3gpp server install|uninstall systemd|launchd. See docs/web-server.md for the full guide.

Database Configuration

Configuration is read from a closed allowlist of environment variables (see ALLOWED_ENV_VARS for the canonical list), the .env file (only the allowlisted vars are honoured), and the TOML config file (everything else).

Plus the bootstrap var DOC3GPP_CONFIG (path to a TOML config file or directory) — see the TOML section below. Any other DOC3GPP_* env var is silently ignored; configure those values via TOML instead.

The remaining settings (cache.size_limit_mb, cache.purge_confirm, tdoc_parse.max_batch, tdoc_parse.max_ftp_depth, sync.*, output.*, db_auto_migrate, http_max_retries, http_retry_backoff, …) are TOML-only — see the example file.

Examples:

# sqlite (omit DOC3GPP_DATABASE_URL to use the pydantic default,
# which resolves to ~/.local/share/doc3gpp/doc3gpp.db)
DOC3GPP_DATABASE_URL=sqlite+pysqlite:////absolute/path/to/doc3gpp.db

Configuration File (TOML)

For structured settings — DB URL plus fetch knobs and per-command output defaults — drop a TOML file at one of these locations (first hit wins):

1. The path named by DOC3GPP_CONFIG (file or directory; absolute or relative). DOC3GPP_CONFIG is independent of the ALLOWED_ENV_VARS allowlist and is the canonical way to pin a config file location from the shell.
2. ./doc3gpp.toml (project-local — check into git for team defaults).
3. ~/.config/doc3gpp/config.toml (user-wide; honors $XDG_CONFIG_HOME).

See the packaged TOML template for the full schema. Highlights:

[output]
format = "json"         # default for every `* list --format`

[output.fields]
meeting = [
  "meeting_id", "name", "location", "start_date",
  "end_date", "ftp_url", "start_doc", "end_doc",
]
tdoc = [
  "tdoc_id", "meeting_name", "title", "source", "type",
  "status", "cr_cat", "spec", "version", "related_wis",
]
tsg = ["tsg_name", "short_name", "description"]
wi  = ["wi_id", "acronym", "release", "name"]

[cache]
dir = "~/.cache/doc3gpp/tdocs"
size_limit_mb = 1024
purge_confirm = true

[spec_doc]
cache_dir = "~/.cache/doc3gpp/specs"
max_zip_size_kb = 0
max_chunk_chars = 1500
chunk_overlap = null
auto_index_on_parse = true
auto_embed_on_parse = true
# Order: (text, sections, tables, spec_id, version, release)
bm25_weights = [5.0, 5.0, 5.0, 1.0, 1.0, 1.0]

# TDoc ZIP archive limits are TOML-only. The byte cap is aggregate
# uncompressed supported `.docx` content per archive; 0 disables either cap.
[tdoc_parse]
max_batch = 100
max_ftp_depth = 2
max_documents_per_zip = 100
max_uncompressed_size_kb = 102400

[search]
enabled = true                       # sqlite-only FTS5 + BM25 search; off = no-op
auto_index_on_parse = true           # keep the index in sync after every parse
snippet_tokens = 8                   # FTS5 snippet() length; --snippet-tokens overrides
# Per-column BM25 weights — drive FTS5 ranking and snippet selection.
# Existing eight-weight values are accepted with a ninth `content_text` weight of 1.0.
# Order: title, ftp_url, meeting_title, meeting_location, wis,
# cover_text, change_text, ttcn_text, content_text.
bm25_weights = [5.0, 0.0, 0.0, 1.0, 5.0, 5.0, 5.0, 5.0, 1.0]

[semantic_search]
enabled = true                       # master switch for vector/hybrid search
# Hybrid blend weight; used by every request that supplies both --text and
# --semantic. It is not a request-level CLI/HTTP/MCP parameter.
fts5_weight = 0.5
# The remaining settings control embedding and index maintenance.
auto_embed_on_parse = true
embedding_base_url = "http://localhost:11434/v1"
embedding_model = "embeddinggemma:300m"
chunk_size = 512
chunk_overlap = 24
rrf_k = 60
fanout_multiplier = 8
max_chunks_per_tdoc = 8              # cap on metadata chunks per TDoc
max_content_chunks_per_tdoc = 16     # TOML-only cap on content chunks per TDoc (CR annexes, family documents)

# Web server + MCP — both TOML-only (no env overrides); only loaded
# with the `doc3gpp[web]` extra installed.
[server]
enabled = true                       # master switch; set false to disable `server` commands
host = "127.0.0.1"
port = 13999

[mcp]
enabled = true                       # mount /mcp; no effect unless server.enabled
transport = "streamable_http"        # streamable_http (default) | sse
allowed_origins = ["http://127.0.0.1", "http://localhost"]  # browser origins allowed to call /mcp
sse_queue_size = 100                 # per-session event queue length

Precedence (highest wins): CLI flag > environment variable > config file > built-in default. Inspect what's in effect with:

doc3gpp config path   # which file is being read
doc3gpp config show   # the fully-resolved settings, as JSON

Edit values without hand-editing the TOML:

doc3gpp config init                       # bootstrap a config file with full defaults
doc3gpp config set sync.auto_sync true    # then edit individual keys

Architecture

tdoc parse batches (database IDs, FTP folders, local folders) and queued web jobs share one service-level pipeline: bounded parallel preparation ([pipeline] prepare_workers) plus a single serialized database writer, with deterministic discovery/output order. DOCX conversion runs in a separate process pool sized by [pipeline] docx_workers; 0 or 1 converts in-process. TDoc content is routed by type through one registry (models/tdoc_content_families.py): services/tdoc_cr_service.py parses CR TDocs, services/tdoc_family_service.py parses WID, LS, and discussion TDocs, services/tdoc_content_service.py serves raw Markdown, and services/tdoc_type_detection.py detects the type for direct mode. Selection rules shared by the CLI, web, and MCP live in services/tdoc_parse_selection.py. Pure conversion lives in parsers/tdoc_zip_convert.py and parsers/tdoc_family_main.py, and content rows are written by storage/repositories/tdoc_content_sql.py. See docs/architecture.md.

The codebase is split into strict layers to keep concerns separate:

See docs/architecture.md for the full design document and module map.

Testing

pytest

SQLite-only profile (excludes online markers):

python -m pytest -q --cov=src/doc3gpp --cov-report=term-missing -m "not online"

Equivalent helper script:

./scripts/test_sqlite.sh

Online tests (opt-in, hits live 3gpp.org and FTP):

python -m pytest -q -m online -rs

Roadmap

Known constraints are documented in AGENTS.md §Known Constraints, and the TDoc extraction pipeline's current state (the R5s / R5w URL templates are verified; the R5- / C6- templates are intentionally unresolved) and the calendar parser's coupling to the current DynaReport layout are called out in docs/architecture.md §Out of scope (today).

Documentation

• Architecture
• CLI reference
• Web server + MCP
• 3GPP knowledge base

Contributing

Issues and pull requests are welcome. There is no formal CONTRIBUTING.md yet — for now:

1. Open an issue describing the change before sending a non-trivial PR.
2. Match the existing style: Python 3.10+, ruff (line-length = 100), strict type hints, layered architecture.
3. Add or update tests in tests/unit/ (mock external calls) and tests/integration/ (sqlite).
4. Keep README.md, AGENTS.md, and docs/*.md in sync when CLI or public-API behavior changes.

License

MIT — Copyright © 2026 jerry wang.

Acknowledgments

• The 3GPP community for making meeting calendars, TDocs, and WI lists publicly available.
• The maintainers of httpx, SQLAlchemy, Pydantic, Typer, BeautifulSoup, openpyxl, tomli_w, and python-docx — the libraries this project stands on.

Support

• Bug reports and feature requests: GitHub Issues
• Source: github.com/jerrywang121/doc3gpp