3GPP Crawler
A command-line tool for crawling the 3GPP FTP server, caching 3GPP document metadata in a local database, and querying structured data via JSON/YAML output.
Github repository: https://github.com/jr2804/3gpp-crawler
Features
- Crawl 3GPP FTP Server: Automatically retrieve TDoc links from RAN, SA, and CT working groups
- Spec Catalog: Crawl and query normalized 3GPP Technical Specifications (TS/TR) from multiple sources
- Local SQLite Database: Store TDoc and Spec metadata for fast querying
- Persistent HTTP Caching: 50-90% faster incremental crawls with automatic request caching
- Case-Insensitive Queries: Search for TDocs regardless of case
- Multiple Output Formats: Export results as table, JSON, or YAML
- Incremental Updates: Only fetch new data on subsequent crawls
- Rich CLI: Beautiful terminal output with progress indicators
- Tag System: Organize TDocs and specs with lightweight tags (no workspace needed)
- MCP Server: Run as stdio MCP server for AI agent integrations (
3gpp-crawler mcp)
Installation
Install as uv tool (recommended)
uv tool install git+https://github.com/jr2804/3gpp-crawler.git
uvx tdoc-crawler --help
Using uv
# Install from source
cd 3gpp-crawler
uv sync
Using pip (not recommended)
# Note: package name may differ from repository name
pip install 3gpp-crawler
Configuration
Environment Variables
Configure the application using a .env file:
# Copy the example file
cp .env.example .env
# Edit .env and add your settings:
# ETSI Online (EOL) credentials (optional for portal authentication)
# Acts as a "3GPP-compatible fallback". While whatthespec.net is the primary
# data source, it is community-maintained. Credentials allow falling back to
# official 3GPP portal endpoints if the primary source is unavailable.
TDC_EOL_USERNAME=your_username
TDC_EOL_PASSWORD=your_password
# HTTP Cache Configuration (optional - uses defaults if not set)
HTTP_CACHE_TTL=7200 # Cache TTL in seconds (default: 7200 = 2 hours)
Alternatively, you can:
# Pass credentials via CLI options:
uvx tdoc-crawler crawl-meetings --eol-username your_username --eol-password your_password
# Configure HTTP caching via CLI:
uvx tdoc-crawler crawl --cache-ttl 3600 --cache-refresh
# Or set environment variables directly:
export TDC_EOL_USERNAME=your_username
export TDC_EOL_PASSWORD=your_password
export HTTP_CACHE_TTL=3600
NOTE: If no credentials are provided, the tool will prompt you interactively when needed.
Quick Start
1. Command Reference
3gpp-crawler is the main CLI; tdoc-crawler and spec-crawler are shortcuts for the tdoc and spec groups.
| Command | Shortcut | Purpose |
|---|---|---|
| 3gpp-crawler | ||
tdoc {subcommand} |
tdoc-crawler |
TDocs and meetings (see below) |
spec {subcommand} |
spec-crawler |
Technical specifications (see below) |
config {init,show,validate,docs} |
Manage configuration | |
tag {add,remove,list} |
Manage lightweight tags on TDocs and specs | |
database {clear-tdocs,clear-specs,clear-meetings,clear-all,export-tags,import-tags} |
db |
Database maintenance: selective/all clears, tag backup & restore |
mcp |
Run as MCP server for AI agents | |
| tdoc group | ||
tdoc crawl |
ct |
Crawl TDoc metadata from FTP |
tdoc crawl-meetings |
cm |
Populate meeting database (Run this first!) |
tdoc query |
qt |
Search for TDocs (auto-fetches if missing) |
tdoc query-meetings |
qm |
Search and display meeting metadata |
tdoc open |
Download and open a TDoc | |
tdoc checkout |
Batch download TDocs | |
tdoc stats |
View database statistics | |
| spec group | ||
spec crawl |
Crawl technical specification metadata | |
spec query |
Search technical specifications | |
spec open |
Download and open latest spec document | |
spec checkout |
Batch download specifications |
1. Crawl Metadata
Gather metadata from 3GPP and WhatTheSpec:
# Populate meeting database (REQUIRED first step)
tdoc-crawler crawl-meetings
# Crawl TDoc metadata (RAN, SA, CT)
tdoc-crawler crawl
# Populate spec catalog
spec-crawler crawl
2. Query Metadata
Search and filter stored information:
# Query a specific TDoc (auto-fetches metadata if missing)
tdoc-crawler query R1-2400001
# Query specifications
spec-crawler query 23.501
# List recent meetings
tdoc-crawler query-meetings --limit 10
3. Utilities & File Access
Open documents, batch download (checkout), and check database status:
# Download and open a TDoc with system default app
tdoc-crawler open R1-2400001
# Download and open latest version of a spec
spec-crawler open 23.501
# Batch download (checkout) TDocs to local folder
tdoc-crawler checkout R1-2400001 S2-2400567
# Batch checkout specifications
spec-crawler checkout 26130-26140
# View database statistics
tdoc-crawler stats
Documentation
For detailed documentation, including command deep-dives, configuration, and architecture, see the Documentation Index.
Outlook
Future enhancements may include:
- Improved RAG with PageIndex, see: https://github.com/VectifyAI/PageIndex.git, https://github.com/VectifyAI/pageindex-mcp
- Web-based UI for interactive querying and document exploration
Development
For information on setting up the development environment, running tests, and code quality standards, please refer to the Development Guide.
License
This project is licensed under the terms specified in the LICENSE file.
Repository initiated with fpgmaas/cookiecutter-uv.
Metadata
Release files for 3gpp-crawler 2026.11.10
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| 3gpp_crawler-2026.11.10.tar.gz | 466.4 kB | With provenance | Details for 3gpp_crawler-2026.11.10.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| 3gpp_crawler-2026.11.10-py3-none-any.whl | Python 3 | none | any | Details for 3gpp_crawler-2026.11.10-py3-none-any.whl |
Total release size: 645.4 kB
Release files / 3gpp_crawler-2026.11.10.tar.gz
| Download URL | 3gpp_crawler-2026.11.10.tar.gz |
|---|---|
| Size | 466.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
af9c0b3fe5ce531a20a2c6a25a884785f1e18a80d6883b1aa28517ede7bf4ac6
|
|
BLAKE2b-256 checksum How to use checksums |
e8127efc67c818e697de22696601b1abd087bcb79de30a2b79662991c0e0c17f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.
Transparency logRelease files / 3gpp_crawler-2026.11.10-py3-none-any.whl
| Download URL | 3gpp_crawler-2026.11.10-py3-none-any.whl |
|---|---|
| Size | 179.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ba28711aac295cbca36f64ffe605d6ab78a7f1e9c07ab69c9874869179c83b1b
|
|
BLAKE2b-256 checksum How to use checksums |
e87f2c6aaec740178293ef149740e38ab6bdf263b32f823b76e0023a8627763e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.
Transparency logSecurity for 3gpp-crawler 2026.11.10
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for 3gpp_crawler-2026.11.10.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.