pygcprofiler 0.3.6

See Python's garbage collector in action without getting in its way. Zero-overhead GC monitoring for production applications.

pygcprofiler

See Python's garbage collector in action without getting in its way.

A zero-overhead GC monitoring tool designed for production applications. Monitor garbage collection events, identify performance bottlenecks, and get actionable optimization recommendations—all without affecting your application's behavior.

✨ Key Features

• Zero Runtime Overhead: Callback only records timestamps and counters—no I/O, no memory checks during GC
• Non-Intrusive: Never modifies GC thresholds, never forces collections, purely observational
• Production-Ready: All output buffered until shutdown, no background threads
• Rich Diagnostics: ASCII flame graphs, threshold alerts, AI-powered optimization prompts
• Flexible Output: Human-readable or JSON format, file logging, configurable alerts

📦 Installation

# From PyPI (when published)
pip install pygcprofiler

# From source
pip install -e .

# With development dependencies
pip install -e ".[dev]"

Requirements: Python 3.12+ and psutil (installed automatically)

🚀 Quick Start

Basic Usage

# Monitor any Python script (all pygcprofiler flags must come BEFORE the script)
pygcprofiler run [pygcprofiler-flags...] your_script.py [script-args...]

# Or use the alias
gc-monitor run [pygcprofiler-flags...] your_script.py [script-args...]

# Pass arguments to your script
pygcprofiler run server.py --port 8000 --debug

With Web Frameworks

# Uvicorn (ASGI)
pygcprofiler run -m uvicorn main:app --host 0.0.0.0 --port 8000

# Gunicorn (WSGI)
pygcprofiler run -m gunicorn app:app --workers 4 --bind 0.0.0.0:8000

# Flask development server
pygcprofiler run app.py

# Django
pygcprofiler run manage.py runserver

Production Monitoring

# JSON output for log aggregation (ELK, Splunk, etc.)
pygcprofiler run --json --log-file gc-events.json -m uvicorn main:app

# Minimal output, only show summary at shutdown
pygcprofiler run --stats-only -m uvicorn main:app

# Custom alert threshold (default: 50ms)
pygcprofiler run --alert-threshold-ms 100 -m uvicorn main:app

Flame Graph Visualization

# ASCII flame graph in terminal
pygcprofiler run --terminal-flamegraph --terminal-flamegraph-color app.py

# Export for external visualization tools
pygcprofiler run --flamegraph-file gc-flame.txt app.py

# Custom bucket size for time grouping
pygcprofiler run --terminal-flamegraph --flamegraph-bucket 10 app.py

📊 Example Output

GMEM Monitoring initialized (Zero Runtime Overhead)
GMEM Running: python -c '...' your_script.py

GMEM GC STOP  | Gen: 0 | Duration: 0.2ms | Collected: 156 | Uncollectable: 0
GMEM GC STOP  | Gen: 0 | Duration: 0.3ms | Collected: 203 | Uncollectable: 0
GMEM GC STOP  | Gen: 1 | Duration: 1.2ms | Collected: 1024 | Uncollectable: 0
GMEM ALERT | Gen 2 pause 52.3ms exceeded 50.0ms threshold
GMEM GC STOP  | Gen: 2 | Duration: 52.3ms | Collected: 15234 | Uncollectable: 0

=== GC MONITORING SUMMARY ===
Total GC collections: 847
Total GC time: 312.5ms
Average GC duration: 0.4ms
Maximum GC duration: 52.3ms

Collections by generation:
  Generation 0: 789 collections
  Generation 1: 52 collections
  Generation 2: 6 collections

=== GC THRESHOLD RECOMMENDATIONS ===
- Generation 2 average pause 28.4ms. Consider reducing long-lived allocations...

⚙️ Command Line Options

Option	Default	Description
--interval	5.0	Snapshot interval in seconds
--json	false	Output in JSON format
--stats-only	false	Only show summary, not individual events
--dump-objects	false	Dump object type analysis at shutdown
--dump-garbage	false	Show uncollectable objects (enables DEBUG_SAVEALL)
--log-file	none	Write output to file
--alert-threshold-ms	50.0	Alert when GC pause exceeds this (ms)
--flamegraph-file	none	Write flame graph data to file
--flamegraph-bucket	5.0	Time bucket size for flame graph (seconds)
--terminal-flamegraph	false	Show ASCII flame graph in terminal
--terminal-flamegraph-width	80	Width of terminal flame graph
--terminal-flamegraph-color	false	Use ANSI colors in flame graph
--duration-buckets	1,5,20,50,100	GC pause duration buckets (ms)
--prompt	false	Generate and display AI optimization prompt at shutdown

🔧 Programmatic Usage

You can also use pygcprofiler as a library:

from gc_monitor.monitor import GCMonitor

# Start monitoring
monitor = GCMonitor(
    alert_threshold_ms=100.0,
    stats_only=True
)

# Your application code here
run_my_application()

# Stop and get results
monitor.stop_monitoring()

🎯 Design Principles

pygcprofiler follows strict zero-runtime-interference principles:

What We Do ✅

gc.callbacks.append(monitor._gc_callback)  # Pure observation
time.perf_counter()  # High-precision timing

What We Never Do ❌

gc.set_threshold(...)  # Never modify GC behavior
gc.collect()           # Never force collections
gc.freeze()            # Never freeze objects
gc.get_objects()       # Never scan object graphs during runtime
print(...)             # Never do I/O in callbacks

Performance Characteristics

• < 0.1% runtime overhead — callback is just timestamp recording
• Zero additional GC pressure — no temporary objects in callbacks
• Same latency profile — no I/O or expensive operations during runtime
• Identical memory usage — no allocations that could affect GC behavior

🔍 Troubleshooting

"Script file not found"

Ensure the script path is correct and accessible:

# Use absolute path
pygcprofiler run /full/path/to/script.py

# Or run from the script's directory
cd /path/to/project && pygcprofiler run script.py

Module mode not working

For modules, use -m as the first argument after run:

# Correct
pygcprofiler run -m uvicorn main:app

# Wrong
pygcprofiler run uvicorn main:app

No GC events showing

If your application doesn't allocate enough objects to trigger GC:

# Lower the snapshot interval
pygcprofiler run app.py --interval 1

# Or run with object dump to see current state
pygcprofiler run app.py --dump-objects

High memory usage in logs

For high-throughput applications, reduce log verbosity:

# Only show summary at shutdown
pygcprofiler run -m uvicorn main:app --stats-only

# Or increase alert threshold to reduce noise
pygcprofiler run -m uvicorn main:app --alert-threshold-ms 200

🔐 Security Considerations

• Log files may contain timing information that could reveal application behavior patterns
• Object dumps (--dump-objects) show type names but not object contents
• Never log to world-readable locations in production
• Use --stats-only in sensitive environments to minimize data exposure

# Secure production setup
pygcprofiler run -m uvicorn main:app \
    --stats-only \
    --log-file /var/log/myapp/gc.log \
    --json

📈 Interpreting Results

GC Generations

• Gen 0: Short-lived objects, collected frequently (< 1ms typical)
• Gen 1: Objects that survived Gen 0, collected less often
• Gen 2: Long-lived objects, collected rarely but takes longest

Warning Signs

Metric	Concern Level	Action
Gen 2 > 10% of collections	🟡 Medium	Consider gc.freeze() after init
Max pause > 50ms	🟠 High	Tune thresholds, reduce allocations
GC CPU > 5%	🔴 Critical	Major optimization needed
Uncollectable > 0	🟡 Medium	Check for reference cycles

Recommended Optimizations

import gc

# After application initialization
gc.collect(2)  # Full collection
gc.freeze()    # Freeze startup objects

# Tune thresholds for your workload
gc.set_threshold(50000, 10, 10)  # Reduce collection frequency

🏗️ Architecture

src/gc_monitor/
├── __init__.py      # Package metadata
├── __main__.py      # CLI entry point
├── cli.py           # Argument parsing
├── codegen.py       # Injection code generation
├── monitor.py       # Core GC monitoring (zero-overhead)
├── logging.py       # Event logging utilities
├── stats.py         # Statistics and recommendations
├── flamegraph.py    # Flame graph rendering
├── memory.py        # Memory utilities
└── prompts.py       # AI optimization prompts

🧪 Development

# Clone the repository
git clone https://github.com/IntegerAlex/pygcprofiler.git
cd pygcprofiler

# Install with dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run linter
ruff check src/ tests/

# Type checking
mypy src/

📄 License

This project is licensed under the GNU Lesser General Public License v2.1 (LGPL-2.1).

You may use this library in proprietary applications, but modifications to the library itself must be shared under the same license. See LICENSE for details.

pygcprofiler - Python Garbage Collection Profiling Tool
Copyright (C) 2024 Akshat Kotpalliwar

This library is free software; you can redistribute it and/or modify it
under the terms of the GNU Lesser General Public License as published by
the Free Software Foundation; either version 2.1 of the License, or
(at your option) any later version.

🤝 Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

📝 Changelog

See CHANGELOG.md for version history and release notes.