MLflow Dependency Analyzer

Smart dependency analysis and minimal requirements generation for MLflow models.

Automatically detect and generate minimal code_paths and requirements lists for your MLflow models using safe AST-based analysis. Ensure portable and reproducible model deployments without dependency bloat.

🚀 Features

• 🔍 Unified Analysis: Complete dependency analysis combining requirements and code paths
• 🧠 Smart Detection: Uses Python's importlib and inspect for accurate module resolution
• 🔒 Safe Analysis: AST-based import discovery - no code execution required
• 📦 MLflow Integration: Built-in support for MLflow's production utilities
• 🎯 Minimal Dependencies: Intelligent pruning eliminates unnecessary packages
• 🔄 Recursive Discovery: Follows deep dependency chains automatically
• 🛡️ Robust Error Handling: Graceful handling of circular dependencies and import errors
• ⚡ Production Ready: Comprehensive test coverage with real-world scenarios

📦 Installation

pip install mlflow-dep-analyzer

🎯 Quick Start

Simple Model Analysis

from mlflow_dep_analyzer import analyze_model_dependencies

# Analyze a single model file
result = analyze_model_dependencies("model.py")

print("📦 External packages needed:")
print(result["requirements"])

print("📂 Local files needed:")
print(result["code_paths"])

MLflow Integration

import mlflow
import mlflow.sklearn
from mlflow_dep_analyzer import analyze_model_dependencies
from sklearn.ensemble import RandomForestClassifier

# Train your model
model = RandomForestClassifier()
# ... training code ...

# Analyze dependencies
deps = analyze_model_dependencies("model.py")

# Log with minimal dependencies
with mlflow.start_run():
    mlflow.sklearn.log_model(
        model,
        "classifier",
        code_paths=deps["code_paths"],
        pip_requirements=deps["requirements"]
    )

📚 API Reference

The MLflow Dependency Analyzer provides a simple, unified interface for dependency analysis:

Main Interface

from mlflow_dep_analyzer import analyze_model_dependencies

# Analyze a single model file
result = analyze_model_dependencies("model.py")

# Analyze with explicit repo root
result = analyze_model_dependencies("model.py", repo_root="/path/to/project")

# Result structure
{
    "requirements": ["pandas", "scikit-learn"],  # External packages to install
    "code_paths": ["model.py", "utils.py"],      # Local files to include
    "analysis": {
        "total_modules": 15,
        "external_packages": 2,
        "local_files": 2,
        "stdlib_modules": 11
    }
}

Class-Based Interface

For advanced use cases or multiple analyses:

from mlflow_dep_analyzer import UnifiedDependencyAnalyzer

# Create analyzer instance
analyzer = UnifiedDependencyAnalyzer(repo_root=".")

# Analyze multiple entry points
result = analyzer.analyze_dependencies(["model.py", "train.py", "utils.py"])

Convenience Functions

from mlflow_dep_analyzer import get_model_requirements, get_model_code_paths

# Get just the requirements list
packages = get_model_requirements("model.py")
# Returns: ["pandas", "scikit-learn", "numpy"]

# Get just the code paths list
files = get_model_code_paths("model.py")
# Returns: ["model.py", "utils.py", "preprocessing.py"]

🏗️ Architecture

The library uses a single, unified analyzer that provides complete dependency analysis:

┌─────────────────────────────┐
│   UnifiedDependencyAnalyzer │
│    (Complete Analysis)      │
└─────────────────────────────┘
                │
                ├─── AST parsing (safe import discovery)
                ├─── importlib.import_module() (dynamic imports)
                ├─── inspect.getsourcefile() (accurate file paths)
                ├─── Smart classification:
                │    ├─── Standard library → ignored
                │    ├─── External packages → requirements
                │    └─── Local files → code_paths + recursive analysis
                └─── MLflow-compatible output

🔍 How It Works

1. AST Parsing: Safely extracts import statements without executing code
2. Module Resolution: Uses importlib.import_module() + inspect.getsourcefile()
3. Smart Classification: Automatically categorizes modules:
   • 📦 External packages → Added to requirements
   • 🐍 Standard library → Ignored (built into Python)
   • 📁 Local files → Added to code_paths and analyzed recursively
4. Dependency Discovery: Recursively follows imports to build complete dependency graph
5. Path Optimization: Generates minimal file lists and package requirements

🌟 Advanced Usage

Complex Project Structure

from mlflow_dep_analyzer import UnifiedDependencyAnalyzer

# Analyze a complex project with src/ structure
analyzer = UnifiedDependencyAnalyzer(repo_root="/path/to/project")
result = analyzer.analyze_dependencies([
    "src/models/classifier.py",
    "src/models/preprocessor.py",
    "src/utils/data_loader.py"
])

print(f"Found {result['analysis']['total_modules']} total modules")
print(f"External packages: {result['analysis']['external_packages']}")
print(f"Local files: {result['analysis']['local_files']}")

Advanced Analysis

from mlflow_dep_analyzer import UnifiedDependencyAnalyzer

# Get detailed analysis results
analyzer = UnifiedDependencyAnalyzer(repo_root=".")
result = analyzer.analyze_dependencies(["model.py"])

# Access detailed metrics
print(f"Total modules found: {result['analysis']['total_modules']}")
print(f"External packages: {result['analysis']['external_packages']}")
print(f"Local files: {result['analysis']['local_files']}")
print(f"Standard library modules: {result['analysis']['stdlib_modules']}")

Error Handling

from mlflow_dep_analyzer import analyze_model_dependencies

try:
    result = analyze_model_dependencies("model.py")
except FileNotFoundError:
    print("Model file not found")
except ImportError as e:
    print(f"Import resolution failed: {e}")

🧪 Examples

See the examples/ directory for complete working examples:

• Basic Usage: Complete MLflow integration demo
• MLflow Integration: Real-world MLflow projects
• Complex Projects: Multi-file analysis with auto-logging

🛠️ Development

Setup

This project uses uv for dependency management:

git clone https://github.com/andrewgross/mlflow-dep-analyzer
cd mlflow-dep-analyzer
uv sync

Running Tests

# Run all tests
uv run pytest

# Run with coverage
uv run pytest --cov=src/mlflow_dep_analyzer --cov-report=html

# Run specific test categories
uv run pytest tests/test_unified_analyzer.py -v

Code Quality

# Linting and formatting
uv run ruff check
uv run ruff format

# Type checking
uv run mypy src/

# Pre-commit hooks
uv run pre-commit run --all-files

Requirements

• Python: 3.8+ (developed with 3.11.11 for Databricks Runtime 15.4 LTS compatibility)
• Core dependencies: MLflow 2.0+
• Development: pytest, ruff, mypy, pre-commit

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

Quick Contribution Guide

1. Fork the repository
2. Create a feature branch: git checkout -b feature-name
3. Make your changes with tests
4. Run the test suite: uv run pytest
5. Submit a pull request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Built on MLflow's production-tested dependency resolution utilities
• Inspired by the need for reliable, minimal MLflow model deployments
• Thanks to the Python AST and importlib developers for robust introspection tools

📈 Roadmap

• Configuration file support
• Plugin system for custom analyzers
• Integration with other ML frameworks
• Dependency vulnerability scanning
• Performance optimizations with caching

---