Skip to main content

3GPP Crawler

3GPP Crawler Logo

A command-line tool for crawling the 3GPP FTP server, caching 3GPP document metadata in a local database, and querying structured data via JSON/YAML output.

Github repository: https://github.com/jr2804/3gpp-crawler

Features

  • Crawl 3GPP FTP Server: Automatically retrieve TDoc links from RAN, SA, and CT working groups
  • Spec Catalog: Crawl and query normalized 3GPP Technical Specifications (TS/TR) from multiple sources
  • Local SQLite Database: Store TDoc and Spec metadata for fast querying
  • Persistent HTTP Caching: 50-90% faster incremental crawls with automatic request caching
  • Case-Insensitive Queries: Search for TDocs regardless of case
  • Multiple Output Formats: Export results as table, JSON, or YAML
  • Incremental Updates: Only fetch new data on subsequent crawls
  • Rich CLI: Beautiful terminal output with progress indicators
  • Tag System: Organize TDocs and specs with lightweight tags (no workspace needed)
  • MCP Server: Run as stdio MCP server for AI agent integrations (3gpp-crawler mcp)

Installation

uv tool install git+https://github.com/jr2804/3gpp-crawler.git
uvx tdoc-crawler --help

Using uv

# Install from source
cd 3gpp-crawler
uv sync
# Note: package name may differ from repository name
pip install 3gpp-crawler

Configuration

Environment Variables

Configure the application using a .env file:

# Copy the example file
cp .env.example .env

# Edit .env and add your settings:

# ETSI Online (EOL) credentials (optional for portal authentication)
# Acts as a "3GPP-compatible fallback". While whatthespec.net is the primary
# data source, it is community-maintained. Credentials allow falling back to
# official 3GPP portal endpoints if the primary source is unavailable.
TDC_EOL_USERNAME=your_username
TDC_EOL_PASSWORD=your_password

# HTTP Cache Configuration (optional - uses defaults if not set)
HTTP_CACHE_TTL=7200                      # Cache TTL in seconds (default: 7200 = 2 hours)

Alternatively, you can:

# Pass credentials via CLI options:
uvx tdoc-crawler crawl-meetings --eol-username your_username --eol-password your_password

# Configure HTTP caching via CLI:
uvx tdoc-crawler crawl --cache-ttl 3600 --cache-refresh

# Or set environment variables directly:
export TDC_EOL_USERNAME=your_username
export TDC_EOL_PASSWORD=your_password
export HTTP_CACHE_TTL=3600

NOTE: If no credentials are provided, the tool will prompt you interactively when needed.

Quick Start

1. Command Reference

3gpp-crawler is the main CLI; tdoc-crawler and spec-crawler are shortcuts for the tdoc and spec groups.

Command Shortcut Purpose
3gpp-crawler
tdoc {subcommand} tdoc-crawler TDocs and meetings (see below)
spec {subcommand} spec-crawler Technical specifications (see below)
config {init,show,validate,docs} Manage configuration
tag {add,remove,list} Manage lightweight tags on TDocs and specs
database {clear-tdocs,clear-specs,clear-meetings,clear-all,export-tags,import-tags} db Database maintenance: selective/all clears, tag backup & restore
mcp Run as MCP server for AI agents
tdoc group
tdoc crawl ct Crawl TDoc metadata from FTP
tdoc crawl-meetings cm Populate meeting database (Run this first!)
tdoc query qt Search for TDocs (auto-fetches if missing)
tdoc query-meetings qm Search and display meeting metadata
tdoc open Download and open a TDoc
tdoc checkout Batch download TDocs
tdoc stats View database statistics
spec group
spec crawl Crawl technical specification metadata
spec query Search technical specifications
spec open Download and open latest spec document
spec checkout Batch download specifications

1. Crawl Metadata

Gather metadata from 3GPP and WhatTheSpec:

# Populate meeting database (REQUIRED first step)
tdoc-crawler crawl-meetings

# Crawl TDoc metadata (RAN, SA, CT)
tdoc-crawler crawl

# Populate spec catalog
spec-crawler crawl

2. Query Metadata

Search and filter stored information:

# Query a specific TDoc (auto-fetches metadata if missing)
tdoc-crawler query R1-2400001

# Query specifications
spec-crawler query 23.501

# List recent meetings
tdoc-crawler query-meetings --limit 10

3. Utilities & File Access

Open documents, batch download (checkout), and check database status:

# Download and open a TDoc with system default app
tdoc-crawler open R1-2400001

# Download and open latest version of a spec
spec-crawler open 23.501

# Batch download (checkout) TDocs to local folder
tdoc-crawler checkout R1-2400001 S2-2400567

# Batch checkout specifications
spec-crawler checkout 26130-26140

# View database statistics
tdoc-crawler stats

Documentation

For detailed documentation, including command deep-dives, configuration, and architecture, see the Documentation Index.

Outlook

Future enhancements may include:

Development

For information on setting up the development environment, running tests, and code quality standards, please refer to the Development Guide.

License

This project is licensed under the terms specified in the LICENSE file.


Repository initiated with fpgmaas/cookiecutter-uv.

Metadata

Release files for 3gpp-crawler 2026.11.12

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for 3gpp-crawler 2026.11.12
File Size Uploaded Provenance Actions
3gpp_crawler-2026.11.12.tar.gz 467.0 kB With provenance Details for 3gpp_crawler-2026.11.12.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for 3gpp-crawler 2026.11.12
File Interpreter ABI Platform Actions
3gpp_crawler-2026.11.12-py3-none-any.whl Python 3 none any Details for 3gpp_crawler-2026.11.12-py3-none-any.whl

Total release size: 646.0 kB

Release files / 3gpp_crawler-2026.11.12.tar.gz

Download URL 3gpp_crawler-2026.11.12.tar.gz
Size 467.0 kB
Tags Source
SHA-256 checksum
How to use checksums
13bc24734e41e5ee0082e7319a98a11e0d5f56a59a68fef9af87ee0800ce1bba
BLAKE2b-256 checksum
How to use checksums
3b933e6e0eb2a879d122821c882815166937a63483dac58e05f812000e94b6d6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

Release files / 3gpp_crawler-2026.11.12-py3-none-any.whl

Download URL 3gpp_crawler-2026.11.12-py3-none-any.whl
Size 179.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
538f02aa9d18bb9e50c6beeec33b86133a47353bdb4c37d47fcdb55140a4a7b4
BLAKE2b-256 checksum
How to use checksums
be3a5b02f73e1c56830b246c0a6506001154ee7d5dd34440ba48a0c71e426878
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

Security for 3gpp-crawler 2026.11.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for 3gpp_crawler-2026.11.12.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page