doc3gpp

Last reviewed: 2026-08-13

Provide 3GPP TDoc information with full text search capability — a Python CLI/library, a local web UI, and an MCP server for AI clients.

Description

doc3gpp scrapes 3GPP meeting calendars, work items (WIs), TDocs, and specifications (TS/TR) from 3gpp.org and persists them to a relational database for programmatic access. It ships in three forms:

• a Python library (SDK) for embedding the data in your own code,
• a Typer-based CLI (doc3gpp) for scripting and ad-hoc queries, and
• a local web server with a built-in browser UI (FastAPI + HTMX + Jinja2) plus a Model Context Protocol endpoint (/mcp) that exposes the same data to AI clients with byte-for-byte JSON parity against the HTTP ?format=json routes.

SQLite is the sole storage backend.

Table of Contents

• Features
• Installation
• Quick Start
• CLI Usage
• Database Configuration
• Configuration File (TOML)
• Architecture
• Testing
• Roadmap
• Documentation
• Contributing
• License
• Acknowledgments
• Support

Features

• Meeting / TDoc / WI / Spec sync — fetch the 3GPP Meetings, TDocs, Work Items List, and 3GPP specifications (TS/TR) with versions. Spec sync fans out across per-spec detail pages in a thread pool and caches the result for spec list / spec show.
• RAN5 testcase sync — fetch the latest TTCN CR Agreement Status snapshot (testcase sync) and query it via testcase list / testcase show (per-path ttcn_status / gcf_ptcrb).
• TDoc extraction — Download and parse TDoc CRs into structured records, and store the Markdown of CR annexes and WID, LS, and discussion TDocs for raw retrieval and search.
• Unified search and indexing — TDoc and spec-document search support FTS5, semantic, hybrid, and filter-only modes; companion index commands maintain the FTS5 and vector components.
• Hybrid semantic search (FTS5 + embeddings) — vector KNN + FTS5 keyword search, merged via reciprocal-rank fusion.
• Web server + MCP — a single-port HTTP server (FastAPI + HTMX + Jinja2) for browsing and searching 3GPP data in a browser, plus a Streamable-HTTP Model Context Protocol endpoint (/mcp) exposing the same data to AI clients with byte-for-byte JSON parity with the HTTP ?format=json routes. Background jobs (sync, parse, unified search-index maintenance, cache purge) run on a shared asyncio worker with live SSE progress.
• SQLite storage backend — via SQLAlchemy 2.0.

Installation

SDK (library)

pip install doc3gpp

Use the SDK to access 3GPP data programmatically:

from doc3gpp.services.meetings_service import MeetingService
from doc3gpp.storage.repositories.meeting_sql import SQLAlchemyMeetingRepository

service = MeetingService(SQLAlchemyMeetingRepository())
meetings = service.list_recent(limit=10)

TDoc detail records (TDocShowRecord, TDocShowRecordByUrl) and content responses expose the outer-ZIP inventory as zip_files with the same null / empty-list / ordered-name semantics as the CLI, HTTP, and MCP surfaces.

CLI (command-line tool)

pip install "doc3gpp[cli]"
# or, for an isolated install:
pipx install "doc3gpp[cli]"

The [cli] extra adds the doc3gpp command (Typer-based subcommands).

Development

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
cp .env.example .env
doc3gpp db init
doc3gpp db check

The [dev] extra includes [cli], pytest, pytest-cov, and ruff.

Optional extras

pip install "doc3gpp[extract]"    # Word and presentation extraction (python-docx, python-pptx)
pip install "doc3gpp[search]"     # FTS5 + BM25 full-text search
pip install "doc3gpp[semantic]"   # Hybrid FTS5 + embedding vector search (sqlite-vec; remote embeddings API)
pip install "doc3gpp[web]"        # Web server + MCP (FastAPI, uvicorn, Jinja2 + HTMX, markdown-it-py)
pip install "doc3gpp[all]"        # every runtime extra: CLI, extraction, search, semantic, web

The [extract] extra installs python-docx and python-pptx>=1.0.2; the PowerPoint library is imported lazily. Legacy Word .doc parsing additionally requires LibreOffice Writer, and legacy .ppt parsing requires headless LibreOffice Impress, with libreoffice or soffice on PATH on the machine running doc3gpp. Each legacy conversion uses a private temporary directory and isolated LibreOffice profile. Both use [word_conversion] timeout_seconds in TOML (default 300 seconds); the allowlisted DOC3GPP_WORD_CONVERSION__TIMEOUT_SECONDS environment variable overrides it. Modern .docx and .pptx members do not require LibreOffice.

Install LibreOffice for legacy Office files

Install LibreOffice on the machine running doc3gpp, including Writer for .doc and Impress for .ppt:

Linux (Debian/Ubuntu):

sudo apt update
sudo apt install libreoffice-writer libreoffice-impress
command -v soffice  # or: command -v libreoffice

On other distributions, install the Writer and Impress packages using your distribution's package manager; package names may differ.

Windows: Download and run the official LibreOffice installer, including Writer and Impress. Add the installation's program directory to your user or system Path in Windows Environment Variables (normally C:\Program Files\LibreOffice\program), then open a new PowerShell window and verify:

Get-Command soffice.exe

Alternatively, for the current PowerShell session only, run $env:Path += ";C:\Program Files\LibreOffice\program" before the verification command and before starting doc3gpp. Adjust the path if LibreOffice was installed elsewhere. Restart an already-running doc3gpp server so it inherits the updated Path. The Python extra is still required: pip install "doc3gpp[extract]".

Quick Start

SDK

from doc3gpp.services.meetings_service import MeetingService
from doc3gpp.services.tdoc_service import TDocService
from doc3gpp.services.wi_service import WiService
from doc3gpp.storage.repositories.meeting_sql import SQLAlchemyMeetingRepository
from doc3gpp.storage.repositories.tdoc_sql import SQLAlchemyTDocRepository
from doc3gpp.storage.repositories.wi_sql import SQLAlchemyWiRepository

meetings = MeetingService(SQLAlchemyMeetingRepository())
tdocs = TDocService(SQLAlchemyTDocRepository())
wis = WiService(SQLAlchemyWiRepository())

recent = meetings.list_recent(limit=5)
for m in recent:
    print(m.meeting_id, m.name, m.end_date)

CLI

doc3gpp db init                            # create schema + seed tsgs table
doc3gpp meeting sync --tsg r5              # scrape DynaReport, validate --tsg
doc3gpp meeting list --limit 5
doc3gpp meeting list --tdoc R5-260013       # find the meeting containing a TDoc
doc3gpp tdoc sync --meeting-id 85434       # requires a stored meeting row
doc3gpp tdoc sync                          # sync every tracked meeting_id in tdocs
doc3gpp tdoc list --tdoc 'R5%'
doc3gpp tdoc parse --meeting-id 85434      # extract CR cover pages; prompts before batch
doc3gpp tdoc parse --tdoc 'R5s26%' --yes   # pattern match, skip confirmation
doc3gpp wi sync --tsg r5                   # scrape WI DynaReport for R5
doc3gpp wi list --release "Rel-19" --limit 50
doc3gpp spec sync --tsg r5                 # scrape spec list + parallel detail pages
doc3gpp spec sync --spec-id 36.579-5       # sync a single stored spec
doc3gpp spec list --type TS --limit 20
doc3gpp spec show 36.579-5                 # header + version rows
doc3gpp spec doc parse --spec 38.331         # fetch, convert, chunk, and index a spec zip
doc3gpp testcase sync                      # latest TTCN status History zip → testcases
doc3gpp testcase list --group 5G --limit 20
doc3gpp testcase show --testcase TC_1      # header + per-path status rows

CLI Usage

The CLI ships twelve sub-apps. The most common entry points are meeting sync (DynaReport calendar), tdoc sync (TDoc-list XLSX + auxiliary file scan), tdoc parse (extract CR covers and ZIP document content), tdoc show --format raw (render converted Markdown for supported .docx, .doc, .pptx, and .ppt ZIP members), spec sync (3GPP specification records with versions), spec doc parse (specification document conversion and chunking), and tdoc search / spec doc search (unified FTS5, semantic, hybrid, and filter-only search).

db — database lifecycle

doc3gpp db init                # create schema + seed tsgs table
doc3gpp db check               # verify connectivity
doc3gpp db reset --yes         # destructive: wipe + recreate SQLite schema

Before using an existing database with a release that adds the TDoc ZIP inventory, run doc3gpp db init --scope main. This additive, idempotent bootstrap creates the new table without resetting existing data or bulk backfilling old ZIPs; their zip_files stays unknown until a supported TDoc is re-parsed with --force.

tsg — 3GPP TSG reference

doc3gpp tsg list               # show the canonical 3GPP TSG list
doc3gpp tsg show --tsg r5      # show a single TSG record
doc3gpp tsg seed               # re-seed the reference table
doc3gpp tsg schema --format json  # describe every tsgs column (meaning, format, values)

meeting — 3GPP meeting calendar

doc3gpp meeting sync --tsg r5              # scrape DynaReport; --tsg validated against tsgs
doc3gpp meeting list --limit 20
doc3gpp meeting list --tdoc R5-260013      # find the meeting whose start_doc/end_doc brackets a TDoc
doc3gpp meeting schema --format json       # describe every meetings column

tdoc — list, parse, show

# sync — every tracked meeting_id or one specific meeting
doc3gpp tdoc sync                                # sync every distinct meeting_id in tdocs
doc3gpp tdoc sync --meeting-id 85434
doc3gpp tdoc sync --meeting "R5--TTCN Workshop#74"

# list — 18 filter flags combine freely
doc3gpp tdoc list --limit 10
doc3gpp tdoc list --tdoc 'R5%'                   # LIKE pattern on tdoc_id
doc3gpp tdoc list --meeting-id 85434 --cr-cat F
doc3gpp tdoc list --tdoc 'R5%' --meeting "%RAN3%"
doc3gpp tdoc list --title '!%Sidelink%'          # NOT LIKE
doc3gpp tdoc list --parsed true                  # only TDocs with a complete extraction (parsed column)

# parse (DB mode) — every flag is a filter
doc3gpp tdoc parse --meeting-id 85434            # CR, WID, LS, and discussion TDocs (default); prompts before batch (pending only)
doc3gpp tdoc parse --tdoc 'R5s26%' --yes         # LIKE pattern; non-interactive
doc3gpp tdoc parse --meeting-id 85434 --meeting '%RAN5%' --cr-cat F
doc3gpp tdoc parse --meeting-id 85434 --release 'Rel-19' --cr-num not-null
doc3gpp tdoc parse --meeting-id 85434 --force    # re-extract everything (includes already-parsed)

# parse (direct mode) — bypasses DB filters
doc3gpp tdoc parse --from-path ~/Downloads/R5s260009.docx                # local .docx → stdout
doc3gpp tdoc parse --from-path ~/Downloads/R5s260009.doc                 # legacy .doc (LibreOffice required)
doc3gpp tdoc parse --from-url https://www.3gpp.org/ftp/.../R5s260009.zip # 3GPP URL → cache + DB
doc3gpp tdoc parse --from-url https://example.com/some.zip --format json -o /tmp/out.json  # non-3GPP URL → in-memory only
doc3gpp tdoc parse --from-path ./tdocs --output ./parsed --recursive --format json         # local batch
doc3gpp tdoc parse --from-url https://www.3gpp.org/ftp/.../Docs/ --recursive --output ./parsed  # online batch
# 3GPP URLs also accept bare www.3gpp.org/ftp/...; HTTPS is required,
# and .docx/.doc/.zip basenames must contain a TDoc id (including RP-... plenary ids).

# show — --tdoc and --ftp-url are mutually exclusive
# The JSON payload includes the outer-ZIP `zip_files` inventory.
doc3gpp tdoc show --tdoc R5s260009 --format json -o r5s260009.json
doc3gpp tdoc show --tdoc R5s260009 --format raw  -o r5s260009.md    # all converted ZIP documents
doc3gpp tdoc show --tdoc R5s260009 --format raw --source-file 'Annex/Support.docx' -o support.md
doc3gpp tdoc show --tdoc R5s260009 --format raw --source-file 'Slides/Overview.pptx' -o slides.md
doc3gpp tdoc show --ftp-url tsg_ran/WG5/.../R5s260009.zip            # URL-keyed lookup
doc3gpp tdoc show --ftp-url https://www.3gpp.org/ftp/.../R5s260009.zip --format raw

# schema — describe every column of the eleven tdoc tables (106 rows, no filters)
doc3gpp tdoc schema --format json

TDoc and spec-document ZIP parsing converts .docx, .doc, .pptx, and .ppt members, including mixed-format archives. Presentations are supported only inside ZIP packages; this does not add standalone presentation parsing or make otherwise ineligible TDoc types eligible. PDF, spreadsheets, images, charts, embedded objects, and nested ZIPs are not converted. Presentation Markdown contains slide titles/text, grouped text, tables, and existing speaker notes; there is no OCR, graphical inference, or presentation-derived CR/spec metadata. Spec slides are searchable content but do not create TOC, section, or table metadata. Original member names and paths are preserved.

For a CR TDoc, the structural Word CR cover remains the main document and alone owns the cover and optional TTCN/body-change sidecars. Presentations are always supporting CR annexes, even if their text resembles a CR cover. WID, LS, and discussion TDocs store their main document and every annex. A failed annex is skipped with a warning that names it, is listed in skipped_sources, and leaves the extraction incomplete until a later normal parse converts it. Use --source-file with --format raw to select one full member path; without it, multiple bodies are emitted as named blocks. tdoc_files remains the metadata for auxiliary files discovered in meeting FTP folders—it is not the inventory of files inside a ZIP.

If the optional extraction dependency or required LibreOffice component is missing, or conversion fails, the error names the original source. A failed main document fails the TDoc; failed annexes retain the existing skip-and-retry behavior. TDoc ZIP downloads retain the max_tdoc_size_kb = 1000 default cap. Inside a TDoc archive, all four supported formats count toward max_documents_per_zip = 100 and the aggregate max_uncompressed_size_kb = 102400 limit; 0 disables either archive limit. The spec ZIP cap is spec_doc.max_zip_size_kb = 0 by default (unlimited). Re-parse already-completed archives with --force to include supported members that earlier versions ignored. Spec-document docx_count retains its historical field name but counts .docx, .doc, .pptx, and .ppt members.

zip_files is a separate, metadata-only inventory for CR, WID new/revised, LS in/out, and discussion TDocs. It lists every non-directory entry in the outer ZIP, regardless of extension, in archive order, retaining duplicates and the original decoded names. Inventory reads member names only: it does not extract members, interpret names as paths, or open nested ZIPs. null means the inventory is unknown; [] means a known empty archive. This is distinct from both tdoc_files auxiliary FTP metadata and the converted documents returned by content reads. A successfully read inventory is stored independently of later parsing/conversion failures and derived-cache eviction.

TDoc-ID detail reads follow the row's current tdoc.ftp_url; an exact --ftp-url lookup can show a historical URL inventory without changing the current TDoc URL. tdoc show table, full/compact Markdown, and JSON output render the inventory separately; HTTP detail pages and JSON, MCP get_tdoc / get_tdoc_content, and the SDK detail/content DTOs expose the same data. The raw Markdown body and exact --source-file selection are unchanged. When the inventory is unknown, the force-parse hint is shown only for a supported TDoc type: use doc3gpp tdoc parse --tdoc <id> --force, or for a supported URL record doc3gpp tdoc parse --from-url <url> --force. A URL with no matching TDoc type has no valid inventory-refresh hint. --force re-parses existing entries and re-downloads a cached ZIP when its provenance cannot be verified. The optional CLI sync.auto_sync metadata behavior for tdoc show --tdoc is unchanged; inventory lookup itself never downloads or parses, and URL detail reads do not auto-sync.

Content is stored in SQL under the single tdoc_extracts parent for each ZIP URL, in one table per store: tdoc_cr_annexes for CR annexes, and tdoc_wid_contents, tdoc_ls_contents, and tdoc_discussion_contents for family documents. The CR main Markdown is never stored in SQL; it lives only in the derived Markdown archive. That archive and its manifest are derived cache artifacts: purging or evicting them does not remove stored content or prevent an FTS5 rebuild, but raw CR main Markdown is unavailable until the TDoc is re-downloaded. Legacy plain-text, gzip, and single-member Markdown cache artifacts remain readable as incomplete version-1 content; parse discovery treats those extractions as pending so the ordinary filtered parse workflow can upgrade them. A URL-anchored raw read is a snapshot and does not implicitly download or extract; the TDoc-ID raw path retains its extraction-on-read behavior except when an exact selected presentation already belongs to a complete extraction, which is read SQL-first after cache eviction.

wi — Work items

doc3gpp wi sync --tsg r5                       # scrape the WI DynaReport page for R5
doc3gpp wi list --limit 10                     # default fields: wi_id, acronym, release, name
doc3gpp wi list --tsg R5 --release "Rel-19" --limit 100
doc3gpp wi schema --format json                # describe every wis column

spec — 3GPP specifications (TS / TR)

# sync — fetch list page + parallel detail pages (TSGs / WI links / ETSI PDF / CR list)
doc3gpp spec sync --tsg r5                          # 24h skip rule via sync.spec_sync_interval
doc3gpp spec sync --tsg r5 --force                  # bypass the skip rule
doc3gpp spec sync --spec-id 36.579-5                # sync a single stored spec (no list page)
doc3gpp spec sync --spec-id 36.579-5 --force        # bypass the skip rule for one spec
doc3gpp spec sync --tsg r5 --per-version-details    # always re-fetch ETSI PDF + CR list per version (default OFF)

# list — filters combine freely (rich-filter grammar: %, !pattern, null, not-null;
# --parsed accepts exactly true or false)
doc3gpp spec list --limit 20
doc3gpp spec list --tsg R5 --type TS --status "Under change control"
doc3gpp spec list --spec-id '36.579%' --title '%conformance%'
doc3gpp spec list --tsg R5 --format json -o r5_specs.json
doc3gpp spec list --parsed true --format json
doc3gpp spec list --parsed false --limit 20

# show — dotted spec id; renders header + version rows
doc3gpp spec show 36.579-5
doc3gpp spec show 36.579-5 --format json -o 36_579-5.json

# schema — describe every column of specs + spec_versions (20 rows, no filters)
doc3gpp spec schema --format json

spec sync honours sync.spec_sync_interval (default 24h) on a per-spec basis: each per-worker _sync_one_spec short-circuits specs whose own specs.last_synced_at is within the interval, and re-syncs the rest; --force bypasses the check. Each spec's last_synced_at is stamped on a successful re-sync so a spec list / spec show always reads from cache. Per-version follow-ups (ETSI PDF + CR list) are skipped by default; pass --per-version-details to fetch them. The default preserves any previously-cached pdf_url / crs values on existing rows.

The default spec list fields end with rapporteurs, parsed. Parsed status is derived from non-null spec_doc_sources.parsed_at values in the separate specdata database; it is not a column in the main specs or spec_versions tables and is not part of spec schema. List JSON returns a comma-separated, numeric-newest-first parsed-version string or native null; table and Markdown output display -. The --parsed true|false filter is applied before pagination. spec show --format json returns native boolean parsed values on every version row. spec show --format json also appends zip_files to each version object: an ordered JSON array that preserves duplicate member names, [] for a known empty ZIP, and null when the inventory is unknown. Table, Markdown, and compact Markdown output show a labeled inventory for each selected version; unknown inventories include a forced-reparse command and a doc3gpp db init --scope specdata schema-upgrade hint. spec show has no --parsed option; that filter belongs to spec list.

SpecService.list_versions batch-enriches selected versions from the separate spec_doc_zip_inventory table. Inventory is collected only while an actual spec-document parse prepares a ZIP; spec sync, schema initialization, detail reads, and immutable parsed-version skips do not collect or backfill it. The inventory lives in specdata rather than a spec_versions column, and spec show reads it from SQL without inspecting the ZIP cache or using the network, so cache eviction does not remove a stored inventory. Existing databases need the additive doc3gpp db init --scope specdata; an unknown parsed version can then be refreshed with doc3gpp spec doc parse --spec <id> --version <v> --force (which re-downloads and re-parses the ZIP).

spec doc — specification document corpus

# use newest version with a published .zip, convert supported Word/presentation members, chunk, and auto-index
doc3gpp spec doc parse --spec 38.331 --release Rel-18 --version 18.5.0
doc3gpp spec doc parse --spec 38.331 --force  # re-download and re-parse

# inspect the stored TOC and search chunk text/combined metadata
doc3gpp spec doc toc show --spec 38.331 --version 18.5.0 --format json
# read a full presentation by its exact ZIP member path (raw Markdown by default)
doc3gpp spec doc show --spec 38.331 --version 18.5.0 --source-file slides/overview.pptx
doc3gpp spec doc search --text "handover" --spec 38.331 --sections "%handover%" --tables "%UE%" --limit 20
doc3gpp spec doc search --semantic "handover" --spec 38.331 --limit 20
doc3gpp spec doc index --rebuild
doc3gpp spec doc schema --format json

Background parse_spec_docs jobs retain the selected spec's failure reason in the job result (failure_details) and log. Each job targets one spec; when both release and version are supplied, their release numbers must align (for example, Rel-18 with 18.5.0).

Spec-document rows live in a separate sibling database, normally <main-stem>_specdata.db, and are cached below the dedicated ~/.cache/doc3gpp/specs root by default. The spec doc schema describes five specdata tables and 34 schema rows, including spec_doc_zip_inventory and the SQL-backed spec_doc_presentations full-file store. Existing installations can add the presentation table without resetting data via doc3gpp db init --scope specdata. The TDoc extraction cache remains ~/.cache/doc3gpp/tdocs. Chunk sections and tables values are newline-delimited combined identifier/title entries; TOC entries retain their section_no and title fields. Section identifiers may be alphanumeric, such as 7.2A.3 or 7.2A.3A. Version selection is numeric rather than lexical; parsed (spec_id, version) rows are immutable and skip on later parses unless --force is supplied, which re-downloads and re-parses. The spec doc show command reads full presentation Markdown from the SQL corpus by exact source path; it does not fetch or parse a ZIP or read the Markdown cache. parse command fetches the ZIP when it is missing and reports ok, skipped, and failed buckets per requested spec. See docs/cli.md for all filters and defaults.

Existing parsed versions are not silently backfilled with presentation rows. After doc3gpp db init --scope specdata, re-parse each version that needs full presentation details with doc3gpp spec doc parse --spec <id> --version <v> --force; --force re-downloads and reparses the ZIP. There is no separate presentation repair route or job.

testcase — RAN5 conformance testcases

# sync — latest TTCN CR Agreement Status History zip (file-identity skip)
doc3gpp testcase sync
doc3gpp testcase sync --force            # re-download + re-parse even when recorded

# list — 10 filter flags combine freely (rich-filter grammar: %, !pattern, null, not-null)
doc3gpp testcase list --limit 20
doc3gpp testcase list --group 5G --spec '38.523-1'
doc3gpp testcase list --status Approved --format json -o testcases.json

# show — one testcase id; renders header + per-path status rows
doc3gpp testcase show --testcase TC_1
doc3gpp testcase show --testcase TC_1 --format json -o tc_1.json

# schema — describe every column of the three testcase tables (21 rows, no filters)
doc3gpp testcase schema --format json

testcase list --status / --gcf-status match ttcn_status / gcf_ptcrb on ANY path (EXISTS); list JSON emits flat per- (testcase_id, group) objects with a nested statuses list of {path, gcf_ptcrb, ttcn_status} objects (no testcase envelope, no group in status rows; None stays null in JSON). Default columns testcase_id,title,spec,group,release,statuses (TOML [output.fields] testcase). testcase show JSON emits the same flat per-(testcase_id, group) shape with all 8 header fields inline plus the nested statuses list.

tdoc search — unified TDoc search

The command accepts independent --text and --semantic inputs. Text-only search uses FTS5, semantic-only search uses the vector index, both inputs use hybrid RRF search, and neither input runs filter-only source SQL.

# FTS5 text search
doc3gpp tdoc search --text "NB-IoT scheduling" --tsg RAN1 --limit 10
doc3gpp tdoc search --text "R5-1234567" --format json
# FTS5 can find text present only in a CR annex or family document
doc3gpp tdoc search --text "annex-specific handover parameter" --format json

# vector search
doc3gpp tdoc search --semantic "power saving for NB-IoT UEs" --limit 10

# hybrid search; [semantic_search].fts5_weight controls the blend
doc3gpp tdoc search --text "NB-IoT" --semantic "power saving" --limit 10

# filter-only source SQL
doc3gpp tdoc search --tsg RAN1 --spec 38.300 --since 2026-01-01

# status and maintenance for both indexes
doc3gpp tdoc index
doc3gpp tdoc index --rebuild --resume --batch 1000
doc3gpp tdoc index --rebuild-all

The auto-index and auto-embed hooks keep the indexes fresh after successful parses when enabled in doc3gpp.toml.

TDoc text syntax

tdoc search --text uses the FTS5 rich-text search pattern. Plain text is wrapped as a quoted expression (implicit AND between terms), while queries with FTS5 operators pass through unchanged:

doc3gpp tdoc search --text "scheduling AND (NR OR LTE)"
doc3gpp tdoc search --text "R5-*"                       # prefix match on TDoc ids
doc3gpp tdoc search --text "38.300*"                    # prefix match on spec ids
doc3gpp tdoc search --text '\"CSI report\" NEAR/5 feedback'

Search JSON contract

CLI JSON, HTTP JSON, and MCP search rows share the same flattened shape. The array is already in final rank order. search_mode is one of fts5, semantic, hybrid, or filter; score is respectively the FTS5 BM25 score, vector distance, RRF score, or JSON null for filter mode. previews is an object only for FTS5 rows and JSON null for the other modes. TDoc rows also include type, status, and best_chunk_id. Public rows do not expose per-source rank fields, nested hits, or separate distance/RRF fields. Every TDoc row also contains matched_documents: an array of {document_id, ftp_url, source_file, score, previews} source matches. FTS5 and hybrid results may include matches (hybrid preserves the FTS5 source list); semantic-only and filter-only rows always contain an empty array. Use each entry's exact ftp_url and full source_file for a source-specific content read; the top-level ftp_url retains its canonical-parent meaning. Stored content documents (CR annexes, and the main and annexes of WID, LS, and discussion TDocs) are FTS5-searchable and semantically chunked, so semantic-only search can match them. The CR main body is not full-text indexed; its cover, change, and TTCN fields are.

The request-level --fts5-weight, --fts5-query, and --sem-query flags were removed. Supply text with --text, semantic input with --semantic, and configure hybrid weighting once with [semantic_search].fts5_weight.

config — TOML config lifecycle

doc3gpp config init                              # bootstrap a TOML config file with full defaults
doc3gpp config path                              # which file is in effect (or "(no config file found)")
doc3gpp config show                              # fully-resolved Settings as JSON for diffing
doc3gpp config set sync.auto_sync true           # write one setting into the active TOML config
doc3gpp config set sync.auto_sync true --dry-run # preview the resulting TOML without writing

cache — Local extraction cache

doc3gpp cache status                  # file count, total bytes, limit, per-subdir breakdown
doc3gpp cache purge --yes             # delete cached markdown sidecars (default scope)
doc3gpp cache purge --scope zips --yes # only the 3GPP-served zip blobs
doc3gpp cache purge --scope all --yes  # both subtrees

Common output options

Every * list command accepts --format {table,json,markdown} and -o/--output PATH. meeting list, tdoc list, and tsg list additionally accept --fields to override the configured column set (wi list and spec list use their configured output.fields.wi / output.fields.spec lists):

doc3gpp tdoc list --format json -o tdocs.json
doc3gpp meeting list --format markdown -o meetings.md
doc3gpp tsg list --format json
doc3gpp wi list --format markdown

tdoc show accepts the same --format + -o/--output pair plus --format raw for the converted Markdown. Raw output emits one document unchanged, or several documents as named source blocks with the main document first. Direct-mode tdoc parse --from-path / --from-url and folder batches also accept --format raw with the same rules.

• --compact — strip output formatting. JSON drops indent and operator space (single line, separators=(",", ":")); Markdown drops CommonMark decorators (bold, italic, headings, bullets, GFM tables, code fences) and emits key: value lines with blank-line section separators. No-op for table and raw. Default: false; opt in globally with [output] compact = true in doc3gpp.toml.

Full command reference: docs/cli.md.

server — web server + MCP

The optional doc3gpp[web] extra installs a single-port HTTP server that serves both a browsable HTMX UI and a Model Context Protocol endpoint. It is enabled by default and binds only to loopback at port 13999; install an OS service if desired, then start it:

doc3gpp server install systemd --no-start     # or `launchd` on macOS
doc3gpp server start                          # opens http://127.0.0.1:13999/

• HTML UI — browse meetings, TDocs, TSGs, WIs, specs, spec-document TOCs, full presentation Markdown, and unified FTS5, semantic, hybrid, and filter-only search results.
• JSON API — every read route accepts ?format=json, byte-for-byte identical to the MCP tools. Unified search is available at GET /tdocs/search and GET /spec-docs/search; index status is available at GET /tdocs/index and GET /spec-docs/index. Full presentation Markdown is available at GET /specs/{spec_id}/docs/content?version=...&source_file=... (format=html by default, or format=json) and as an exact TDoc source at GET /tdocs/{tdoc_id}/content?source_file=...&format=json; these reads use SQL and never download or parse. Index maintenance jobs are enqueued with POST /jobs/tdocs/index and POST /jobs/spec-docs/index. Raw TDoc Markdown is served by GET /tdocs/{tdoc_id}/content and GET /tdocs/by-url/content?ftp_url=... (format=markdown by default, or format=json); these reads never download. Removed /search/sem routes are not current APIs.
• MCP — http://127.0.0.1:13999/mcp exposes the same reads and jobs, including get_spec_doc_content and get_tdoc_content for exact-path full presentation Markdown. Search tools are search_tdoc and search_spec_docs; index tools are get_tdoc_index, index_tdocs, get_spec_doc_index, and index_spec_docs. Search rows are flattened and JSON-parity is preserved across CLI, HTTP, and MCP.
• Jobs — sync, parse, unified index maintenance, and cache purge run on a shared asyncio worker; watch live progress over SSE at /jobs/{id}/events.

Point an MCP client at the endpoint. For example, in Claude Desktop's claude_desktop_config.json (or any client that supports a Streamable-HTTP MCP server):

{
  "mcpServers": {
    "doc3gpp": {
      "type": "http",
      "url": "http://127.0.0.1:13999/mcp"
    }
  }
}

For clients that only speak the legacy SSE protocol, switch the server transport and point at the SSE endpoint instead:

doc3gpp config set mcp.transport sse

{
  "mcpServers": {
    "doc3gpp": {
      "type": "sse",
      "url": "http://127.0.0.1:13999/mcp/sse"
    }
  }
}

Manage the process with doc3gpp server start|stop|status|logs, doc3gpp server install|uninstall systemd|launchd. See docs/web-server.md for the full guide.

Database Configuration

Configuration is read from a closed allowlist of environment variables (see ALLOWED_ENV_VARS for the canonical list), the .env file (only the allowlisted vars are honoured), and the TOML config file (everything else).

Plus the bootstrap var DOC3GPP_CONFIG (path to a TOML config file or directory) — see the TOML section below. Any other DOC3GPP_* env var is silently ignored; configure those values via TOML instead.

The remaining settings (cache.size_limit_mb, cache.purge_confirm, tdoc_parse.max_batch, tdoc_parse.max_ftp_depth, sync.*, output.*, db_auto_migrate, http_max_retries, http_retry_backoff, …) are TOML-only — see the example file.

Examples:

# sqlite (omit DOC3GPP_DATABASE_URL to use the pydantic default,
# which resolves to ~/.local/share/doc3gpp/doc3gpp.db)
DOC3GPP_DATABASE_URL=sqlite+pysqlite:////absolute/path/to/doc3gpp.db

Configuration File (TOML)

For structured settings — DB URL plus fetch knobs and per-command output defaults — drop a TOML file at one of these locations (first hit wins):

1. The path named by DOC3GPP_CONFIG (file or directory; absolute or relative). DOC3GPP_CONFIG is independent of the ALLOWED_ENV_VARS allowlist and is the canonical way to pin a config file location from the shell.
2. ./doc3gpp.toml (project-local — check into git for team defaults).
3. ~/.config/doc3gpp/config.toml (user-wide; honors $XDG_CONFIG_HOME).

See the packaged TOML template for the full schema. Highlights:

[output]
format = "json"         # default for every `* list --format`

[output.fields]
meeting = [
  "meeting_id", "name", "location", "start_date",
  "end_date", "ftp_url", "start_doc", "end_doc",
]
tdoc = [
  "tdoc_id", "meeting_name", "title", "source", "type",
  "status", "cr_cat", "spec", "version", "related_wis",
]
tsg = ["tsg_name", "short_name", "description"]
wi  = ["wi_id", "acronym", "release", "name"]

[cache]
dir = "~/.cache/doc3gpp/tdocs"
size_limit_mb = 1024
purge_confirm = true

[spec_doc]
cache_dir = "~/.cache/doc3gpp/specs"
max_zip_size_kb = 0
max_chunk_chars = 1500
chunk_overlap = null
auto_index_on_parse = true
auto_embed_on_parse = true
# Order: (text, sections, tables, spec_id, version, release)
bm25_weights = [5.0, 5.0, 5.0, 1.0, 1.0, 1.0]

# TDoc ZIP archive limits are TOML-only. The byte cap is aggregate
# uncompressed `.docx`/`.doc`/`.pptx`/`.ppt` content per archive; 0 disables either cap.
[tdoc_parse]
max_batch = 100
max_ftp_depth = 2
max_documents_per_zip = 100
max_uncompressed_size_kb = 102400

[search]
enabled = true                       # sqlite-only FTS5 + BM25 search; off = no-op
auto_index_on_parse = true           # keep the index in sync after every parse
snippet_tokens = 8                   # FTS5 snippet() length; --snippet-tokens overrides
# Per-column BM25 weights — drive FTS5 ranking and snippet selection.
# Existing eight-weight values are accepted with a ninth `content_text` weight of 1.0.
# Order: title, ftp_url, meeting_title, meeting_location, wis,
# cover_text, change_text, ttcn_text, content_text.
bm25_weights = [5.0, 0.0, 0.0, 1.0, 5.0, 5.0, 5.0, 5.0, 1.0]

[semantic_search]
enabled = true                       # master switch for vector/hybrid search
# Hybrid blend weight; used by every request that supplies both --text and
# --semantic. It is not a request-level CLI/HTTP/MCP parameter.
fts5_weight = 0.5
# The remaining settings control embedding and index maintenance.
auto_embed_on_parse = true
embedding_base_url = "http://localhost:11434/v1"
embedding_model = "embeddinggemma:300m"
chunk_size = 512
chunk_overlap = 24
rrf_k = 60
fanout_multiplier = 8
max_chunks_per_tdoc = 8              # cap on metadata chunks per TDoc
max_content_chunks_per_tdoc = 16     # TOML-only cap on content chunks per TDoc (CR annexes, family documents)

# Web server + MCP — both TOML-only (no env overrides); only loaded
# with the `doc3gpp[web]` extra installed.
[server]
enabled = true                       # master switch; set false to disable `server` commands
host = "127.0.0.1"
port = 13999

[mcp]
enabled = true                       # mount /mcp; no effect unless server.enabled
transport = "streamable_http"        # streamable_http (default) | sse
allowed_origins = ["http://127.0.0.1", "http://localhost"]  # browser origins allowed to call /mcp
sse_queue_size = 100                 # per-session event queue length

Precedence (highest wins): CLI flag > environment variable > config file > built-in default. Inspect what's in effect with:

doc3gpp config path   # which file is being read
doc3gpp config show   # the fully-resolved settings, as JSON

Edit values without hand-editing the TOML:

doc3gpp config init                       # bootstrap a config file with full defaults
doc3gpp config set sync.auto_sync true    # then edit individual keys

Architecture

tdoc parse batches (database IDs, FTP folders, local folders) and queued web jobs share one service-level pipeline: bounded parallel preparation ([pipeline] prepare_workers) plus a single serialized database writer, with deterministic discovery/output order. DOCX conversion runs in a separate process pool sized by [pipeline] docx_workers; 0 or 1 converts in-process. TDoc content is routed by type through one registry (models/tdoc_content_families.py): services/tdoc_cr_service.py parses CR TDocs, services/tdoc_family_service.py parses WID, LS, and discussion TDocs, services/tdoc_content_service.py serves raw Markdown, and services/tdoc_type_detection.py detects the type for direct mode. Selection rules shared by the CLI, web, and MCP live in services/tdoc_parse_selection.py. Pure conversion lives in parsers/tdoc_zip_convert.py and parsers/tdoc_family_main.py, and content rows are written by storage/repositories/tdoc_content_sql.py. See docs/architecture.md.

The codebase is split into strict layers to keep concerns separate:

See docs/architecture.md for the full design document and module map.

Testing

pytest

SQLite-only profile (excludes online markers):

python -m pytest -q --cov=src/doc3gpp --cov-report=term-missing -m "not online"

Equivalent helper script:

./scripts/test_sqlite.sh

Online tests (opt-in, hits live 3gpp.org and FTP):

python -m pytest -q -m online -rs

Roadmap

Known constraints are documented in AGENTS.md §Known Constraints, and the TDoc extraction pipeline's current state (the R5s / R5w URL templates are verified; the R5- / C6- templates are intentionally unresolved) and the calendar parser's coupling to the current DynaReport layout are called out in docs/architecture.md §Out of scope (today).

Documentation

• Architecture
• CLI reference
• Web server + MCP
• 3GPP knowledge base

Contributing

Issues and pull requests are welcome. There is no formal CONTRIBUTING.md yet — for now:

1. Open an issue describing the change before sending a non-trivial PR.
2. Match the existing style: Python 3.10+, ruff (line-length = 100), strict type hints, layered architecture.
3. Add or update tests in tests/unit/ (mock external calls) and tests/integration/ (sqlite).
4. Keep README.md, AGENTS.md, and docs/*.md in sync when CLI or public-API behavior changes.

License

MIT — Copyright © 2026 jerry wang.

Acknowledgments

• The 3GPP community for making meeting calendars, TDocs, and WI lists publicly available.
• The maintainers of httpx, SQLAlchemy, Pydantic, Typer, BeautifulSoup, openpyxl, tomli_w, and python-docx — the libraries this project stands on.

Support

• Bug reports and feature requests: GitHub Issues
• Source: github.com/jerrywang121/doc3gpp