Installation & Quick Start

1. Install via pip (Recommended)

pip install polyglotminer

Both polyglotminer and its short alias pgminer are immediately available in your terminal.

# Verify environment health
polyglotminer doctor

# Launch the interactive terminal wizard (TUI)
polyglotminer

git clone https://github.com/SariYanouche/PolyglotMiner.git
cd PolyglotMiner
python -m venv .venv
source .venv/bin/activate       # On Windows: .\.venv\Scripts\Activate.ps1
pip install -e .

2. Configure Autocompletion & Auth (Optional)

# Enable tab autocompletion for your shell (PowerShell, Bash, Zsh, Fish)
polyglotminer completion --install

# (Optional) High-volume candidate harvesting without GitHub rate-limits (60 -> 5,000 req/hr)
gh auth login   # or: export GITHUB_TOKEN="ghp_..."

Overview & Methodology

PolyglotMiner traverses repository histories without checking out physical file trees or downloading file blobs, achieving mining throughputs exceeding 70+ repositories per minute.

• $O(1)$ Memory Blobless Tree Diffing: Traverses commit trees via native git ls-tree -r streaming with zero blob network calls.
• Native Subprocess Streaming: Direct git rev-list and git log --name-status pipelines eliminate commit-object memory allocation overhead.
• Dual Execution Modes: Fast Empirical Module Mode (default, file-turnover analysis) or Physical LOC Mode (--full-loc, batched git cat-file --batch streaming).
• Noise & False-Positive Elimination: Excludes C/C++/Objective-C header pairings (.h vs .c/.cpp/.m), identical renames/moves, DevOps/CI glue scripts (.sh, .bat, Dockerfiles), and vendor directories (node_modules/, Pods/, vendor/).

Empirical Migration Archetypes

$$\text{Milestone Pearson Correlation: } r = \frac{\sum (s_i - \bar{s})(t_i - \bar{t})}{\sqrt{\sum (s_i - \bar{s})^2 \sum (t_i - \bar{t})^2}} \le -0.25$$

CLI Command Reference

Both polyglotminer and its short alias pgminer are registered executable commands.

Key Command Examples

# 1. Point-in-time profiling (local directory or remote repository)
polyglotminer scan https://github.com/facebook/react --commit v18.2.0 --open

# 2. Longitudinal evolution trajectory across 10 milestone intervals
polyglotminer history handmux/handmux --samples 10 --open

# 3. Directed migration campaign (target specific transitions e.g. C -> C++ or JS -> TS)
polyglotminer campaign -p "C->C++,JS->TS" --workers 4 --limit 50 --open

# 4. Continuous overnight mining loop with stratified star sampling
polyglotminer campaign --min-stars 50 --max-stars 5000 --seed 42 --workers 2 --loop

# 5. Mine multi-language co-changes with minimum LOC delta threshold
polyglotminer mine carlphilipp/chicago-commutes --source-lang Java --target-lang Kotlin --export-csv pairs.csv

# 6. Pairwise 2-commit turnover comparison
polyglotminer compare vuejs/vue v2.6.0 v3.0.0 --open

Web Analytics Studio

Launch the dedicated read-only analytics suite with polyglotminer ui (default: http://localhost:8080).

• Graph Navigator & Spotlight Visualizations: Instant navigation across transition ecosystems, turnover archetypes, star tier distributions, and commit depth brackets.
• Live Heuristic Tuning: Interactive threshold sliders (candidate pairs threshold, minimum confidence score, archetype filtering) with instant client-side recalculation.
• Instant SSR & Skeleton Shimmers: Virtualized tables and lightweight SQLite streaming (<10MB RAM) for lag-free exploration of large datasets.
• Academic Export Suite: 1-click export of publication-ready high-resolution figures (300 DPI PNG), LaTeX booktabs tables, and CSV datasets.
• REST API: Integrates /api/summary, /api/corpus, /api/repositories, /api/repositories/saved, /api/cochanges, and dynamic /<owner>/<repo>/evolution_latest.html routing.

Database Architecture

Analytical state is stored in an embedded SQLite database (reports/polyglotminer.db) with Write-Ahead Logging (WAL):

• Process Recycling: Campaign workers recycle processes every 25 repositories (maxtasksperchild=25) for 100% OS memory reclamation.
• Watchdog & Failure Resilience: 75-second per-repository watchdog timeout. Failed or timeout repositories are permanently tracked in SQLite so resume queues never stall.

Verification & Tests

# Run full automated test suite
python -m unittest discover tests

# Run specific verification suites
python -m unittest tests/test_cli_smoke.py
python -m unittest tests/test_directed_campaign.py
python -m unittest tests/test_header_migration_exclusion.py

Repository Structure

PolyglotMiner/
├── polyglotminer/                  # Core research package
│   ├── __init__.py                 # Version and package metadata (v1.0.1)
│   ├── cli.py                      # Unified CLI parser and router
│   ├── scanner.py                  # Single-commit line-counting and tokenizer
│   ├── evolution.py                # Milestone sampler and trajectory engine
│   ├── cochange.py                 # Multi-language commit and snippet extractor
│   ├── campaign.py                 # Multiprocess mass-mining campaign engine
│   ├── cache.py                    # Git clone cache pool (.cache/repos/)
│   ├── languages.py                # Language specifications and color maps
│   ├── config.py                   # Vendor and asset exclusion rules
│   ├── db.py                       # SQLite persistence layer (WAL mode)
│   ├── storage.py                  # Storage orchestration
│   ├── server.py                   # Embedded REST API and Web Studio server
│   ├── doctor.py                   # System diagnostics & dependency check
│   ├── completion.py               # Shell autocompletion generator
│   ├── wizard.py                   # Interactive quick-start terminal wizard (TUI)
│   ├── models.py                   # Analytical data structures
│   ├── reporters/                  # CLI and HTML formatters
│   └── templates/                  # Web Studio Jinja2 templates
├── tests/                          # Automated verification test suite
├── main.py                         # Universal entry point
├── requirements.txt                # Package dependencies
├── setup.py                        # Setuptools build configuration
└── README.md                       # Documentation

Citation

If you use PolyglotMiner in academic publications, please cite:

@software{polyglotminer2026,
  author = {Sari, Yanouche},
  title = {PolyglotMiner: An Empirical Research Platform for Longitudinal Software Evolution and Cross-Language Migration Mining},
  year = {2026},
  version = {1.0.2},
  url = {https://github.com/SariYanouche/PolyglotMiner}
}