---

---

🔬 Motivation & Research Scope

While multi-language software systems are ubiquitous, studying their evolutionary dynamics empirically presents major methodological challenges:

1. Phased Migrations (Strangler Fig Pattern): Developers rarely rewrite and delete legacy code in atomic single-commit operations; migrations span multiple commits, weeks, or release cycles with concurrent co-existence.
2. Structural Restructuring ($1 \to N$ and $N \to 1$): Rewriting across languages frequently involves architectural consolidation (e.g., C/C++ .h + .c pairs into unified Rust/Zig modules) or decomposition (splitting legacy monoliths into modular components).
3. Severe False-Positive Hazards: Multi-language repositories frequently experience polyglot co-evolution, C/C++ header extraction, build reclassifications, and third-party vendoring that distort naive migration metrics.

PolyglotMiner v1.1.0 addresses these challenges through zero-checkout streaming tree traversal, a multi-stage deterministic-to-neural matching waterfall, sliding-window Strangler Fig tracking, and strict false-positive elimination guards.

---

⚡ Quick Start

1. Installation

# Core framework (CPU, O(1) tree traversal, Code DNA & contract invariants)
pip install polyglotminer

# With GPU acceleration (CUDA/MPS dense vector embeddings via PyTorch)
pip install "polyglotminer[gpu]"

Verify your environment and diagnostics:

polyglotminer doctor

2. Immediate Examples

# 1. Profile a repository's current language composition and lines of code
polyglotminer scan facebook/react --open

# 2. Track longitudinal language trajectories across 10 historical milestones
polyglotminer history jackwener/OpenCLI --samples 10 --gpu --open

# 3. Mine commit-level file conversions and refactorings
polyglotminer mine carlphilipp/chicago-commutes --gpu --open

# 4. Launch the local interactive Web Analytics Studio
polyglotminer ui --port 8080

---

🧠 Detection Architecture (v1.1.0)

PolyglotMiner combines high-throughput deterministic matching with semantic recovery in a Hierarchical Waterfall:

[Candidate Commit / Multi-Commit Window: Deleted & Added Files]
                             │
                             ▼
  ┌─────────────────────────────────────────────────────────────┐
  │ Pre-Filter Guards: Exclude Vendored Code, 100% Identical   │
  │ Renames, C/C++ Header Splits, and Test-Prod Cross-Pairings │
  └──────────────────────────────┬──────────────────────────────┘
                                 │
                                 ▼
  ┌─────────────────────────────────────────────────────────────┐
  │ Stage 1: Exact Basename & Header+Impl Consolidation         │ ──► O(1) Deterministic
  └──────────────────────────────┬──────────────────────────────┘
                                 │ (unmatched files)
                                 ▼
  ┌─────────────────────────────────────────────────────────────┐
  │ Stage 2: Normalized Stem (camelCase / snake_case) & Suffixes│ ──► Morphological Norm
  └──────────────────────────────┬──────────────────────────────┘
                                 │ (unmatched files)
                                 ▼
  ┌─────────────────────────────────────────────────────────────┐
  │ Stage 3: Subtoken Jaccard Overlap & Edit Distance           │ ──► Subtoken Similarity
  └──────────────────────────────┬──────────────────────────────┘
                                 │ (unmatched files)
                                 ▼
  ┌─────────────────────────────────────────────────────────────┐
  │ Stage 4: Contract Invariants (API Routes, DB Tables, RPC)   │ ──► Behavioral Invariant
  └──────────────────────────────┬──────────────────────────────┘
                                 │ (unmatched files)
                                 ▼
  ┌─────────────────────────────────────────────────────────────┐
  │ Stage 5: Structural Solvers (1:N Decomposition / N:1 Merge) │ ──► Structural Token Bags
  └──────────────────────────────┬──────────────────────────────┘
                                 │ (unmatched files)
                                 ▼
  ┌─────────────────────────────────────────────────────────────┐
  │ Stage 6: Strangler Fig Tracker (Multi-Commit Windows)       │ ──► Phased Lifecycle Arcs
  └──────────────────────────────┬──────────────────────────────┘
                                 │ (unmatched files & --gpu)
                                 ▼
  ┌─────────────────────────────────────────────────────────────┐
  │ Stage 7: Hardware-Accelerated Dense Neural Embeddings       │ ──► 8,192-token Cosine Sim
  │ (Default: jinaai/jina-embeddings-v2-base-code)             │
  └─────────────────────────────────────────────────────────────┘

Empirical Migration Archetypes

Archetype	Evolutionary Behavior & Diagnostic Criteria
Direct Replacement	Atomic 1-to-1 file translation (file.js $\to$ file.ts) within a single commit.
Strangler Fig Migration	Phased asynchronous replacement across multiple commits ($k \ge 1$) with temporary co-existence.
1-to-$N$ Decomposition	Monolithic source module split into $N$ cohesive target components while preserving token invariants.
$N$-to-1 Consolidation	$N$ source files (e.g. .h + .c or legacy helpers) consolidated into 1 unified target implementation.
Macro Displacement	Longitudinal trajectory where the incoming language displaces the legacy language ($r \le -0.70$).
Heterogeneous Interop	Stable multi-language co-existence across historical milestones without replacement.

---

💻 CLI Command Reference

Both polyglotminer and its alias pgminer provide a modular research CLI:

Command	Alias	Category	Description
history	h	Evolution	Sample milestone checkpoints and compute longitudinal trajectories
mine	m	Mining	Extract multi-language co-change commits and file replacements
campaign	c	Mining	High-throughput multiprocess candidate harvesting across GitHub strata
scan	s	Profiling	Point-in-time language distribution, file counts, and physical LOC
diff	d	Churn	Pairwise 2-commit file turnover, line deltas, and replacements
ui	u	Studio	Launch local interactive web studio dashboard on port 8080
bookmark	b	Datasets	Curate, tag, annotate, and export benchmark research repositories
list	l	Workspace	List stored snapshots, evolution runs, and mined datasets
rebuild	r	Workspace	Rebuild HTML dashboards and reports from SQLite datasets
clean	x	Workspace	Purge stored reports, specific runs, or analysis caches
doctor	k	Diagnostics	Environment diagnostics for Git, Python runtime, and GPU acceleration

Key Research Workflows

1. Longitudinal Trajectory Analysis (history)

# Analyze a target repository across 10 historical milestones
polyglotminer history jackwener/OpenCLI --samples 10 --gpu --open

# Deep physical LOC traversal between specific release tags
polyglotminer history vuejs/vue --from v2.6.0 --to v3.0.0 --full-loc

2. Mining Commit Refactorings (mine)

# Extract multi-language co-change commits with neural matching
polyglotminer mine carlphilipp/chicago-commutes --gpu --open

# Target specific migration transitions with minimum LOC churn threshold
polyglotminer mine . --source-lang Java --target-lang Kotlin --min-loc 10 --export-csv transitions.csv

3. Large-Scale Empirical Discovery Campaigns (campaign)

# Harvest and mine 50 candidate repositories matching specific migration transitions
polyglotminer campaign -p "Python -> Rust, Java -> Kotlin" --limit 50 --workers 4 --gpu

# Continuous unattended empirical harvester with commit, star, and file bounds
polyglotminer campaign --min-stars 50 --min-commits 100 --max-commits 5000 --min-files 100 --workers 4 --no-logs

---

🐍 Python Research API

PolyglotMiner can be integrated directly into Python data pipelines, Jupyter notebooks, or pandas workflows:

from polyglotminer.scanner import scan_repository
from polyglotminer.evolution import analyze_evolution
from polyglotminer.cochange import mine_cochange_commits

# 1. Profile language distribution
snapshot = scan_repository("facebook/react")
print(f"Total lines: {snapshot.total_lines:,}")
for lang, stats in snapshot.languages.items():
    print(f"  {lang:<12} {stats.code_lines:>8,} LOC ({stats.percentage:.1f}%)")

# 2. Mine cross-language file transitions programmatically
transitions = mine_cochange_commits(
    repo_path="jackwener/OpenCLI",
    source_lang="Python",
    target_lang="Rust"
)
for t in transitions:
    for pair in t.migration_pairs:
        print(f"[{t.commit_hash[:7]}] {pair.source_file} -> {pair.target_file} ({pair.match_type})")

---

📊 Interactive Web Studio

Launch the local Zero-Trust web analytics studio at http://localhost:8080:

polyglotminer ui --port 8080

• Longitudinal Visualizations: Interactive dual-axis milestone trajectories, language churn breakdowns, and crossover inflection points.
• Component Restructuring Inspection: Drill down into $1 \to N$ decompositions, $N \to 1$ consolidations, and multi-commit Strangler Fig lifecycle arcs.
• Academic Export Suite: 1-click export of publication-ready LaTeX booktabs tables, high-resolution figures (300 DPI), and raw CSV datasets.
• Zero-Latency Persistence: Backed by embedded SQLite with Write-Ahead Logging (WAL), serving sub-5ms queries across thousands of mined repositories.

---

☁️ Google Colab & Hardware Acceleration

A ready-to-run Google Colab notebook is available with NVIDIA GPU acceleration and Google Drive persistence:

See notebooks/README.md for step-by-step instructions on running campaigns with cloud acceleration.

---

🧪 Verification & Testing

PolyglotMiner maintains an extensive test suite verifying matching passes, Git parsing, database concurrency, and false-positive guards:

# Run full unit test suite
python -m unittest discover tests

# Run specific domain test suites
python -m unittest tests/test_cli_smoke.py
python -m unittest tests/test_strangler_fig.py
python -m unittest tests/test_v117_consolidation_decomposition_fixes.py
python -m unittest tests/test_false_positive_migrations.py

---

📚 Citation

If you use PolyglotMiner in academic research, empirical software engineering studies, or replication packages, please cite:

@software{polyglotminer2026,
  author    = {Sari, Yanouche},
  title     = {PolyglotMiner: An Empirical Research Framework for Software Evolution and Cross-Language Migration Mining},
  year      = {2026},
  version   = {1.1.0},
  url       = {https://github.com/SariYanouche/PolyglotMiner}
}

---

📄 License

This project is licensed under the MIT License.