Skip to main content

3GPP Crawler

3GPP Crawler Banner

PyPI Python Docs License: MIT

A command-line tool for crawling the 3GPP FTP server, caching 3GPP document metadata in a local database, and querying structured data via JSON/YAML output.

Github repository: https://github.com/jr2804/3gpp-crawler

Features

  • Official sources, no account needed: meeting and TDoc metadata come from the 3GPP FTP server and portal — basic crawling and querying work without EOL credentials
  • Spec Catalog: Crawl and query normalized 3GPP Technical Specifications (TS/TR); the official 3GPP source is tried first, WhatTheSpec serves as optional convenience/fallback
  • Local SQLite Database: Store TDoc and Spec metadata for fast querying
  • Persistent HTTP Caching: 50-90% faster incremental crawls with automatic request caching
  • Case-Insensitive Queries: Search for TDocs regardless of case
  • Multiple Output Formats: Export results as table, JSON, or YAML
  • Incremental Updates: Only fetch new data on subsequent crawls
  • Archive Listings: Inspect the files inside TDoc/spec zip archives (tdoc files, spec files) without a full download
  • Rich CLI: Beautiful terminal output with progress indicators
  • Tag System: Organize TDocs and specs with lightweight tags (no workspace needed)
  • MCP Server: Run as stdio MCP server for AI agent integrations (3gpp-crawler mcp)

Installation

Requires Python 3.13 or 3.14.

uv tool install 3gpp-crawler
3gpp-crawler --help

Equivalent alternatives:

pipx install 3gpp-crawler      # pipx
pip install 3gpp-crawler       # pip
uvx 3gpp-crawler --help        # run ad hoc, without installing

From source (development)

git clone https://github.com/jr2804/3gpp-crawler.git
cd 3gpp-crawler
uv sync
uv run 3gpp-crawler --help

Configuration

Environment Variables

Configure the application using a .env file:

# Copy the example file
cp .env.example .env

# Edit .env and add your settings:

# ETSI Online (EOL) credentials (optional - authenticated portal access)
# Single/missing TDocs come from the official 3GPP portal by default;
# WhatTheSpec is only a third-party convenience/fallback. Credentials give
# you official (authenticated) endpoints and full metadata.
TDC_EOL_USERNAME=your_username
TDC_EOL_PASSWORD=your_password

# HTTP Cache Configuration (optional - uses defaults if not set)
HTTP_CACHE_TTL=7200                      # Cache TTL in seconds (default: 7200 = 2 hours)

Alternatively, you can:

# Pass credentials via CLI options:
uvx 3gpp-crawler meetings crawl --eol-username your_username --eol-password your_password

# Or set environment variables directly:
export TDC_EOL_USERNAME=your_username
export TDC_EOL_PASSWORD=your_password
export HTTP_CACHE_TTL=3600

NOTE: Credentials are optional — crawling and querying work against official unauthenticated sources without them. If provided, they are resolved in this order: CLI options > config file > environment variables. To be prompted interactively for missing credentials, set TDC_EOL_PROMPT=true (or credentials.prompt in the config file).

Quick Start

1. Command Reference

3gpp-crawler is the main CLI; tdoc-crawler and spec-crawler are shortcuts for the tdoc and spec groups.

Top-level commands (3gpp-crawler)

Command Shortcut Purpose
tdoc {subcommand} tdoc-crawler TDocs (see below)
spec {subcommand} spec-crawler Technical specifications (see below)
meetings {crawl,query} — 3GPP meetings: crawl meeting metadata, query the meeting database
work-items {subcommand} — 3GPP work item catalogue (DynaReport lists, enrichment, audit)
config {init,show,validate,docs} — Manage configuration
tag {add,remove,list} — Manage lightweight tags on TDocs and specs
database {clear-tdocs,clear-specs,clear-meetings,clear-all,export-tags,import-tags} db Database maintenance: selective/all clears, tag backup & restore
mcp — Run as MCP server for AI agents
version — Show installed version

TDocs (3gpp-crawler tdoc / tdoc-crawler)

Command Alias Purpose
crawl ct Crawl TDoc metadata from FTP
query qt Search for TDocs (auto-fetches if missing)
open — Download and open a TDoc
checkout — Batch download TDocs
main-file {set,clear,show,list,reclassify} — Manage main-document overrides and re-run classification for checked-out archives
files — List the files inside a TDoc's zip archive
stats — View database statistics

Meetings (3gpp-crawler meetings)

Command Purpose
crawl Populate meeting database (run this first)
query Search and display meeting metadata

Specifications (3gpp-crawler spec / spec-crawler)

Command Purpose
crawl Crawl technical specification metadata
query Search technical specifications
open Download and open latest spec document
checkout Batch download specifications
files List the files inside a spec's zip archive

2. Crawl Metadata

Gather metadata from the official 3GPP sources:

# Populate meeting database (REQUIRED first step)
3gpp-crawler meetings crawl

# Crawl TDoc metadata (RAN, SA, CT)
tdoc-crawler crawl

# Populate spec catalog
spec-crawler crawl

3. Query Metadata

Search and filter stored information:

# Query a specific TDoc (auto-fetches metadata if missing)
tdoc-crawler query R1-2400001

# Query specifications
spec-crawler query 23.501

# List recent meetings
3gpp-crawler meetings query --limit 10

4. Utilities & File Access

Open documents, batch download (checkout), and check database status:

# Download and open a TDoc with system default app
tdoc-crawler open R1-2400001

# Download and open latest version of a spec
spec-crawler open 23.501

# Batch download (checkout) TDocs to local folder
tdoc-crawler checkout R1-2400001 S2-2400567

# Batch checkout specifications
spec-crawler checkout 26130-26140

# List the files inside a TDoc/spec archive
tdoc-crawler files R1-2400001
spec-crawler files 26.130

# View database statistics
tdoc-crawler stats

Model Context Protocol (MCP)

Expose the crawler to AI agents via the stdio MCP server:

{
  "mcpServers": {
    "3gpp-crawler": {
      "command": "uvx",
      "args": ["3gpp-crawler", "mcp"]
    }
  }
}

Documentation

For detailed documentation, including command deep-dives, configuration, and architecture, see the Documentation Index or docs/index.md.

Outlook

Future enhancements may include:

Development

For information on setting up the development environment, running tests, and code quality standards, please refer to the Development Guide.

License

This project is licensed under the MIT License.


Repository initiated with fpgmaas/cookiecutter-uv.

Metadata

Release files for 3gpp-crawler 2026.11.24

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for 3gpp-crawler 2026.11.24
File Size Uploaded Provenance Actions
3gpp_crawler-2026.11.24.tar.gz 908.4 kB With provenance Details for 3gpp_crawler-2026.11.24.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for 3gpp-crawler 2026.11.24
File Interpreter ABI Platform Actions
3gpp_crawler-2026.11.24-py3-none-any.whl Python 3 none any Details for 3gpp_crawler-2026.11.24-py3-none-any.whl

Total release size: 1.2 MB

Release files / 3gpp_crawler-2026.11.24.tar.gz

Download URL 3gpp_crawler-2026.11.24.tar.gz
Size 908.4 kB
Tags Source
SHA-256 checksum
How to use checksums
e88401e74a421b7c3bfa5d6e0effe46817d08e7d9d1673f61b49eee1f8c6307a
BLAKE2b-256 checksum
How to use checksums
8a241c77d471cee65ae788a77fccd04cce3973c658dcd3528e170a61ad07ed4b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

Release files / 3gpp_crawler-2026.11.24-py3-none-any.whl

Download URL 3gpp_crawler-2026.11.24-py3-none-any.whl
Size 264.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
ea50417a95243e9e4a4b1f6fdc4d0e727ba03fbaaecbbedabd5cf14b0fe5057f
BLAKE2b-256 checksum
How to use checksums
68c8a90b39bd859128d91fbf76dca4fa0f03c24b43394579c0b25c7fee638bd2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

Security for 3gpp-crawler 2026.11.24

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for 3gpp_crawler-2026.11.24.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page