greenmining

An empirical Python library for Mining Software Repositories (MSR) in Green IT research.

Overview

greenmining is a research-grade Python library designed for empirical Mining Software Repositories (MSR) studies in Green IT. It enables researchers and practitioners to:

• Mine repositories at scale - Search, fetch, and clone GitHub repositories via GraphQL API with configurable filters
• Classify green commits - Detect 124 sustainability patterns from the Green Software Foundation (GSF) catalog using 332 keywords
• Analyze any repository by URL - Direct Git-based analysis with support for private repositories
• Measure energy consumption - RAPL, CodeCarbon, and CPU Energy Meter backends for power profiling
• Carbon footprint reporting - CO2 emissions calculation with 20+ country profiles and cloud region support
• Method-level analysis - Per-method complexity and metrics via Lizard integration
• Generate research datasets - Statistical analysis, temporal trends, and publication-ready reports

Installation

Via pip

pip install greenmining

With energy measurement

pip install greenmining[energy]

From source

git clone https://github.com/adam-bouafia/greenmining.git
cd greenmining
pip install -e .

Quick Start

Pattern Detection

from greenmining import GSF_PATTERNS, is_green_aware, get_pattern_by_keywords

print(f"Total patterns: {len(GSF_PATTERNS)}")  # 124 patterns across 15 categories

commit_msg = "Optimize Redis caching to reduce energy consumption"
if is_green_aware(commit_msg):
    patterns = get_pattern_by_keywords(commit_msg)
    print(f"Matched patterns: {patterns}")

Fetch Repositories

from greenmining import fetch_repositories

repos = fetch_repositories(
    github_token="your_token",
    max_repos=50,
    min_stars=500,
    keywords="kubernetes cloud-native",
    languages=["Python", "Go"],
    created_after="2020-01-01",
    pushed_after="2023-01-01",
)

for repo in repos[:5]:
    print(f"- {repo.full_name} ({repo.stars} stars)")

Clone Repositories

from greenmining import fetch_repositories, clone_repositories

repos = fetch_repositories(github_token="your_token", max_repos=10, keywords="android")

# Clone into ./greenmining_repos/ with sanitized directory names
paths = clone_repositories(repos)
print(f"Cloned {len(paths)} repositories")

Analyze Repositories by URL

from greenmining import analyze_repositories

results = analyze_repositories(
    urls=[
        "https://github.com/kubernetes/kubernetes",
        "https://github.com/istio/istio",
    ],
    max_commits=100,
    parallel_workers=2,
    energy_tracking=True,
    energy_backend="auto",
    method_level_analysis=True,
    include_source_code=True,
    github_token="your_token",
    since_date="2020-01-01",
    to_date="2025-12-31",
)

for result in results:
    print(f"{result.name}: {result.green_commit_rate:.1%} green")

Access Pattern Data

from greenmining import GSF_PATTERNS

# Get patterns by category
cloud = {k: v for k, v in GSF_PATTERNS.items() if v['category'] == 'cloud'}
print(f"Cloud patterns: {len(cloud)}")

# All categories
categories = set(p['category'] for p in GSF_PATTERNS.values())
print(f"Categories: {sorted(categories)}")

Energy Measurement

from greenmining.energy import get_energy_meter, CPUEnergyMeter

# Auto-detect best backend
meter = get_energy_meter("auto")
meter.start()
# ... your workload ...
result = meter.stop()
print(f"Energy: {result.joules:.2f} J, Power: {result.watts_avg:.2f} W")

Statistical Analysis

from greenmining.analyzers import StatisticalAnalyzer, TemporalAnalyzer

stat = StatisticalAnalyzer()
temporal = TemporalAnalyzer(granularity="quarter")

# Pattern correlations, effect sizes, temporal trends
# See experiment notebook for full usage

Metrics-to-Power Correlation

from greenmining.analyzers import MetricsPowerCorrelator

correlator = MetricsPowerCorrelator()
correlator.fit(
    metrics=["complexity", "nloc", "code_churn"],
    metrics_values={
        "complexity": [10, 20, 30, 40],
        "nloc": [100, 200, 300, 400],
        "code_churn": [50, 100, 150, 200],
    },
    power_measurements=[5.0, 8.0, 12.0, 15.0],
)
print(f"Feature importance: {correlator.feature_importance}")

Features

Core Capabilities

• Pattern Detection: 124 sustainability patterns across 15 categories from the GSF catalog
• Keyword Analysis: 332 green software detection keywords
• Repository Fetching: GraphQL API with date, star, and language filters
• Repository Cloning: Sanitized directory names in ./greenmining_repos/
• URL-Based Analysis: Direct Git-based analysis from GitHub URLs (HTTPS and SSH)
• Batch Processing: Parallel analysis of multiple repositories
• Private Repository Support: Authentication via SSH keys or GitHub tokens

Analysis & Measurement

• Energy Measurement: RAPL, CodeCarbon, and CPU Energy Meter backends
• Carbon Footprint Reporting: CO2 emissions with 20+ country profiles (AWS, GCP, Azure)
• Metrics-to-Power Correlation: Pearson and Spearman analysis between code metrics and power
• Method-Level Analysis: Per-method complexity metrics via Lizard integration
• Source Code Access: Before/after source code for refactoring detection
• Process Metrics: DMM size, complexity, interfacing via PyDriller
• Statistical Analysis: Correlations, effect sizes, and temporal trends
• Multi-format Output: JSON, CSV, pandas DataFrame

Energy Backends

GSF Pattern Categories

124 patterns across 15 categories:

Development

git clone https://github.com/adam-bouafia/greenmining.git
cd greenmining
pip install -e ".[dev]"

pytest tests/
black greenmining/ tests/
ruff check greenmining/ tests/

Requirements

• Python 3.9+
• PyGithub, PyDriller, pandas, colorama, tqdm

Optional:

pip install greenmining[energy]      # psutil, codecarbon
pip install greenmining[dev]         # pytest, black, ruff, mypy

License

MIT License - See LICENSE for details.

Links

• GitHub: https://github.com/adam-bouafia/greenmining
• PyPI: https://pypi.org/project/greenmining/
• Documentation: https://greenmining.readthedocs.io/
• Docker Hub: https://hub.docker.com/r/adambouafia/greenmining