Skip to main content

3GPP Crawler

3GPP Crawler Banner

PyPI Python Docs License: MIT

A command-line tool for crawling the 3GPP FTP server, caching 3GPP document metadata in a local database, and querying structured data via JSON/YAML output.

Github repository: https://github.com/jr2804/3gpp-crawler

Features

  • Official sources, no account needed: meeting and TDoc metadata come from the 3GPP FTP server and portal — basic crawling and querying work without EOL credentials
  • Spec Catalog: Crawl and query normalized 3GPP Technical Specifications (TS/TR); the official 3GPP source is tried first, WhatTheSpec serves as optional convenience/fallback
  • Local SQLite Database: Store TDoc and Spec metadata for fast querying
  • Persistent HTTP Caching: 50-90% faster incremental crawls with automatic request caching
  • Case-Insensitive Queries: Search for TDocs regardless of case
  • Multiple Output Formats: Export results as table, JSON, or YAML
  • Incremental Updates: Only fetch new data on subsequent crawls
  • Archive Listings: Inspect the files inside TDoc/spec zip archives (tdoc files, spec files) without a full download
  • Rich CLI: Beautiful terminal output with progress indicators
  • Tag System: Organize TDocs and specs with lightweight tags (no workspace needed)
  • MCP Server: Run as stdio MCP server for AI agent integrations (3gpp-crawler mcp)

Installation

Requires Python 3.13 or 3.14.

uv tool install 3gpp-crawler
3gpp-crawler --help

Equivalent alternatives:

pipx install 3gpp-crawler      # pipx
pip install 3gpp-crawler       # pip
uvx 3gpp-crawler --help        # run ad hoc, without installing

From source (development)

git clone https://github.com/jr2804/3gpp-crawler.git
cd 3gpp-crawler
uv sync
uv run 3gpp-crawler --help

Configuration

Environment Variables

Configure the application using a .env file:

# Copy the example file
cp .env.example .env

# Edit .env and add your settings:

# ETSI Online (EOL) credentials (optional - authenticated portal access)
# Single/missing TDocs come from the official 3GPP portal by default;
# WhatTheSpec is only a third-party convenience/fallback. Credentials give
# you official (authenticated) endpoints and full metadata.
TDC_EOL_USERNAME=your_username
TDC_EOL_PASSWORD=your_password

# HTTP Cache Configuration (optional - uses defaults if not set)
HTTP_CACHE_TTL=7200                      # Cache TTL in seconds (default: 7200 = 2 hours)

Alternatively, you can:

# Pass credentials via CLI options:
uvx tdoc-crawler crawl-meetings --eol-username your_username --eol-password your_password

# Or set environment variables directly:
export TDC_EOL_USERNAME=your_username
export TDC_EOL_PASSWORD=your_password
export HTTP_CACHE_TTL=3600

NOTE: Credentials are optional — crawling and querying work against official unauthenticated sources without them. If provided, they are resolved in this order: CLI options > config file > environment variables. To be prompted interactively for missing credentials, set TDC_EOL_PROMPT=true (or credentials.prompt in the config file).

Quick Start

1. Command Reference

3gpp-crawler is the main CLI; tdoc-crawler and spec-crawler are shortcuts for the tdoc and spec groups.

Command Shortcut Purpose
3gpp-crawler
tdoc {subcommand} tdoc-crawler TDocs and meetings (see below)
spec {subcommand} spec-crawler Technical specifications (see below)
config {init,show,validate,docs} Manage configuration
tag {add,remove,list} Manage lightweight tags on TDocs and specs
database {clear-tdocs,clear-specs,clear-meetings,clear-all,export-tags,import-tags} db Database maintenance: selective/all clears, tag backup & restore
mcp Run as MCP server for AI agents
version Show installed version
tdoc group
tdoc crawl-meetings cm Populate meeting database (Run this first!)
tdoc crawl ct Crawl TDoc metadata from FTP
tdoc query qt Search for TDocs (auto-fetches if missing)
tdoc query-meetings qm Search and display meeting metadata
tdoc open Download and open a TDoc
tdoc checkout Batch download TDocs
tdoc files List the files inside a TDoc's zip archive
tdoc stats View database statistics
spec group
spec crawl Crawl technical specification metadata
spec query Search technical specifications
spec open Download and open latest spec document
spec checkout Batch download specifications
spec files List the files inside a spec's zip archive

2. Crawl Metadata

Gather metadata from the official 3GPP sources:

# Populate meeting database (REQUIRED first step)
tdoc-crawler crawl-meetings

# Crawl TDoc metadata (RAN, SA, CT)
tdoc-crawler crawl

# Populate spec catalog
spec-crawler crawl

3. Query Metadata

Search and filter stored information:

# Query a specific TDoc (auto-fetches metadata if missing)
tdoc-crawler query R1-2400001

# Query specifications
spec-crawler query 23.501

# List recent meetings
tdoc-crawler query-meetings --limit 10

4. Utilities & File Access

Open documents, batch download (checkout), and check database status:

# Download and open a TDoc with system default app
tdoc-crawler open R1-2400001

# Download and open latest version of a spec
spec-crawler open 23.501

# Batch download (checkout) TDocs to local folder
tdoc-crawler checkout R1-2400001 S2-2400567

# Batch checkout specifications
spec-crawler checkout 26130-26140

# List the files inside a TDoc/spec archive
tdoc-crawler files R1-2400001
spec-crawler files 26.130

# View database statistics
tdoc-crawler stats

Model Context Protocol (MCP)

Expose the crawler to AI agents via the stdio MCP server:

{
  "mcpServers": {
    "3gpp-crawler": {
      "command": "uvx",
      "args": ["3gpp-crawler", "mcp"]
    }
  }
}

Documentation

For detailed documentation, including command deep-dives, configuration, and architecture, see the Documentation Index or docs/index.md.

Outlook

Future enhancements may include:

Development

For information on setting up the development environment, running tests, and code quality standards, please refer to the Development Guide.

License

This project is licensed under the MIT License.


Repository initiated with fpgmaas/cookiecutter-uv.

Metadata

Release files for 3gpp-crawler 2026.11.15

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for 3gpp-crawler 2026.11.15
File Size Uploaded Provenance Actions
3gpp_crawler-2026.11.15.tar.gz 836.9 kB With provenance Details for 3gpp_crawler-2026.11.15.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for 3gpp-crawler 2026.11.15
File Interpreter ABI Platform Actions
3gpp_crawler-2026.11.15-py3-none-any.whl Python 3 none any Details for 3gpp_crawler-2026.11.15-py3-none-any.whl

Total release size: 1.1 MB

Release files / 3gpp_crawler-2026.11.15.tar.gz

Download URL 3gpp_crawler-2026.11.15.tar.gz
Size 836.9 kB
Tags Source
SHA-256 checksum
How to use checksums
dd1d8a82e4002de784e88f2f2645e66e5868a5a96dc34e2f4e1af6d96787215a
BLAKE2b-256 checksum
How to use checksums
1ec076436827c4f2f3a545feb79e10e21cdfede4a6612ecb236d0b007473d00b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.

Transparency log

Release files / 3gpp_crawler-2026.11.15-py3-none-any.whl

Download URL 3gpp_crawler-2026.11.15-py3-none-any.whl
Size 223.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
44967a7974af5587e51ba1515bf48c0d1eabbb8f8608000e545c53a535c8cf62
BLAKE2b-256 checksum
How to use checksums
1f5c91c9ea8d4f8ad2eacd05ef07fcbd9107f060cf2a7ed3b523ac41f2bfce29
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.

Transparency log

Security for 3gpp-crawler 2026.11.15

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for 3gpp_crawler-2026.11.15.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 12, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page