3GPP Crawler
A command-line tool for crawling the 3GPP FTP server, caching 3GPP document metadata in a local database, and querying structured data via JSON/YAML output.
Github repository: https://github.com/jr2804/3gpp-crawler
Features
- Official sources, no account needed: meeting and TDoc metadata come from the 3GPP FTP server and portal — basic crawling and querying work without EOL credentials
- Spec Catalog: Crawl and query normalized 3GPP Technical Specifications (TS/TR); the official 3GPP source is tried first, WhatTheSpec serves as optional convenience/fallback
- Local SQLite Database: Store TDoc and Spec metadata for fast querying
- Persistent HTTP Caching: 50-90% faster incremental crawls with automatic request caching
- Case-Insensitive Queries: Search for TDocs regardless of case
- Multiple Output Formats: Export results as table, JSON, or YAML
- Incremental Updates: Only fetch new data on subsequent crawls
- Archive Listings: Inspect the files inside TDoc/spec zip archives (
tdoc files,spec files) without a full download - Rich CLI: Beautiful terminal output with progress indicators
- Tag System: Organize TDocs and specs with lightweight tags (no workspace needed)
- MCP Server: Run as stdio MCP server for AI agent integrations (
3gpp-crawler mcp)
Installation
Requires Python 3.13 or 3.14.
From PyPI (recommended)
uv tool install 3gpp-crawler
3gpp-crawler --help
Equivalent alternatives:
pipx install 3gpp-crawler # pipx
pip install 3gpp-crawler # pip
uvx 3gpp-crawler --help # run ad hoc, without installing
From source (development)
git clone https://github.com/jr2804/3gpp-crawler.git
cd 3gpp-crawler
uv sync
uv run 3gpp-crawler --help
Configuration
Environment Variables
Configure the application using a .env file:
# Copy the example file
cp .env.example .env
# Edit .env and add your settings:
# ETSI Online (EOL) credentials (optional - authenticated portal access)
# Single/missing TDocs come from the official 3GPP portal by default;
# WhatTheSpec is only a third-party convenience/fallback. Credentials give
# you official (authenticated) endpoints and full metadata.
TDC_EOL_USERNAME=your_username
TDC_EOL_PASSWORD=your_password
# HTTP Cache Configuration (optional - uses defaults if not set)
HTTP_CACHE_TTL=7200 # Cache TTL in seconds (default: 7200 = 2 hours)
Alternatively, you can:
# Pass credentials via CLI options:
uvx 3gpp-crawler meetings crawl --eol-username your_username --eol-password your_password
# Or set environment variables directly:
export TDC_EOL_USERNAME=your_username
export TDC_EOL_PASSWORD=your_password
export HTTP_CACHE_TTL=3600
NOTE: Credentials are optional — crawling and querying work against official unauthenticated sources without them. If provided, they are resolved in this order: CLI options > config file > environment variables. To be prompted interactively for missing credentials, set TDC_EOL_PROMPT=true (or credentials.prompt in the config file).
Quick Start
1. Command Reference
3gpp-crawler is the main CLI; tdoc-crawler and spec-crawler are shortcuts for the tdoc and spec groups.
Top-level commands (3gpp-crawler)
| Command | Shortcut | Purpose |
|---|---|---|
tdoc {subcommand} |
tdoc-crawler |
TDocs (see below) |
spec {subcommand} |
spec-crawler |
Technical specifications (see below) |
meetings {crawl,query} |
— | 3GPP meetings: crawl meeting metadata, query the meeting database |
work-items {subcommand} |
— | 3GPP work item catalogue (DynaReport lists, enrichment, audit) |
config {init,show,validate,docs} |
— | Manage configuration |
tag {add,remove,list} |
— | Manage lightweight tags on TDocs and specs |
database {clear-tdocs,clear-specs,clear-meetings,clear-all,export-tags,import-tags} |
db |
Database maintenance: selective/all clears, tag backup & restore |
mcp |
— | Run as MCP server for AI agents |
version |
— | Show installed version |
TDocs (3gpp-crawler tdoc / tdoc-crawler)
| Command | Alias | Purpose |
|---|---|---|
crawl |
ct |
Crawl TDoc metadata from FTP |
query |
qt |
Search for TDocs (auto-fetches if missing) |
open |
— | Download and open a TDoc |
checkout |
— | Batch download TDocs |
main-file {set,clear,show,list,reclassify} |
— | Manage main-document overrides and re-run classification for checked-out archives |
files |
— | List the files inside a TDoc's zip archive |
stats |
— | View database statistics |
Meetings (3gpp-crawler meetings)
| Command | Purpose |
|---|---|
crawl |
Populate meeting database (run this first) |
query |
Search and display meeting metadata |
Specifications (3gpp-crawler spec / spec-crawler)
| Command | Purpose |
|---|---|
crawl |
Crawl technical specification metadata |
query |
Search technical specifications |
open |
Download and open latest spec document |
checkout |
Batch download specifications |
files |
List the files inside a spec's zip archive |
2. Crawl Metadata
Gather metadata from the official 3GPP sources:
# Populate meeting database (REQUIRED first step)
3gpp-crawler meetings crawl
# Crawl TDoc metadata (RAN, SA, CT)
tdoc-crawler crawl
# Populate spec catalog
spec-crawler crawl
3. Query Metadata
Search and filter stored information:
# Query a specific TDoc (auto-fetches metadata if missing)
tdoc-crawler query R1-2400001
# Query specifications
spec-crawler query 23.501
# List recent meetings
3gpp-crawler meetings query --limit 10
4. Utilities & File Access
Open documents, batch download (checkout), and check database status:
# Download and open a TDoc with system default app
tdoc-crawler open R1-2400001
# Download and open latest version of a spec
spec-crawler open 23.501
# Batch download (checkout) TDocs to local folder
tdoc-crawler checkout R1-2400001 S2-2400567
# Batch checkout specifications
spec-crawler checkout 26130-26140
# List the files inside a TDoc/spec archive
tdoc-crawler files R1-2400001
spec-crawler files 26.130
# View database statistics
tdoc-crawler stats
Model Context Protocol (MCP)
Expose the crawler to AI agents via the stdio MCP server:
{
"mcpServers": {
"3gpp-crawler": {
"command": "uvx",
"args": ["3gpp-crawler", "mcp"]
}
}
}
Documentation
For detailed documentation, including command deep-dives, configuration, and architecture, see the Documentation Index or docs/index.md.
Outlook
Future enhancements may include:
- Improved RAG with PageIndex, see: https://github.com/VectifyAI/PageIndex.git, https://github.com/VectifyAI/pageindex-mcp
- Web-based UI for interactive querying and document exploration
Development
For information on setting up the development environment, running tests, and code quality standards, please refer to the Development Guide.
License
This project is licensed under the MIT License.
Repository initiated with fpgmaas/cookiecutter-uv.
Metadata
Release files for 3gpp-crawler 2026.11.21
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| 3gpp_crawler-2026.11.21.tar.gz | 883.5 kB | With provenance | Details for 3gpp_crawler-2026.11.21.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| 3gpp_crawler-2026.11.21-py3-none-any.whl | Python 3 | none | any | Details for 3gpp_crawler-2026.11.21-py3-none-any.whl |
Total release size: 1.1 MB
Release files / 3gpp_crawler-2026.11.21.tar.gz
| Download URL | 3gpp_crawler-2026.11.21.tar.gz |
|---|---|
| Size | 883.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
5cdc6f98e6a3cfadfd8995bc215c4a51a3dce78a55f78c88aa284739c197caa4
|
|
BLAKE2b-256 checksum How to use checksums |
3e52705357f4ea0d784d3b65fffc04a12761cf7b84720d2195bfa5fbfbfc1512
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.
Transparency logRelease files / 3gpp_crawler-2026.11.21-py3-none-any.whl
| Download URL | 3gpp_crawler-2026.11.21-py3-none-any.whl |
|---|---|
| Size | 247.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6410812357c5c9e10293a6f4bd335c6cc38a407eb821b81e4b819674cf456a17
|
|
BLAKE2b-256 checksum How to use checksums |
a02f6fe3ce5489601461d62217fe85ef33a8ef33331bda71640192a9e850ff45
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.
Transparency logSecurity for 3gpp-crawler 2026.11.21
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for 3gpp_crawler-2026.11.21.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.