mlflow-dep-analyzer 0.4.0

Smart dependency analysis and minimal requirements generation for MLflow models

MLflow Dependency Analyzer

Smart dependency analysis and minimal requirements generation for MLflow models.

Automatically detect and generate minimal code_paths and requirements lists for your MLflow models using safe AST-based analysis. Ensure portable and reproducible model deployments without dependency bloat.

🚀 Features

• 🔍 Unified Analysis: Complete dependency analysis combining requirements and code paths
• 🧠 Smart Detection: Uses Python's importlib and inspect for accurate module resolution
• 🔒 Safe Analysis: AST-based import discovery - no code execution required
• 📦 MLflow Integration: Built-in support for MLflow's production utilities
• 🎯 Minimal Dependencies: Intelligent pruning eliminates unnecessary packages
• 🔄 Recursive Discovery: Follows deep dependency chains automatically
• 🛡️ Robust Error Handling: Graceful handling of circular dependencies and import errors
• ⚡ Production Ready: Comprehensive test coverage with real-world scenarios

📦 Installation

pip install mlflow-dep-analyzer

🎯 Quick Start

Simple Model Analysis

from mlflow_dep_analyzer import analyze_model_dependencies

# Analyze a single model file
result = analyze_model_dependencies("model.py")

print("📦 External packages needed:")
print(result["requirements"])

print("📂 Local files needed:")
print(result["code_paths"])

MLflow Integration

import mlflow
import mlflow.sklearn
from mlflow_dep_analyzer import analyze_model_dependencies
from sklearn.ensemble import RandomForestClassifier

# Train your model
model = RandomForestClassifier()
# ... training code ...

# Analyze dependencies
deps = analyze_model_dependencies("model.py")

# Log with minimal dependencies
with mlflow.start_run():
    mlflow.sklearn.log_model(
        model,
        "classifier",
        code_paths=deps["code_paths"],
        pip_requirements=deps["requirements"]
    )

📚 API Reference

The MLflow Dependency Analyzer provides a simple, unified interface for dependency analysis:

Main Interface

from mlflow_dep_analyzer import analyze_model_dependencies

# Analyze a single model file
result = analyze_model_dependencies("model.py")

# Analyze with explicit repo root
result = analyze_model_dependencies("model.py", repo_root="/path/to/project")

# Result structure
{
    "requirements": ["pandas", "scikit-learn"],  # External packages to install
    "code_paths": ["model.py", "utils.py"],      # Local files to include
    "analysis": {
        "total_modules": 15,
        "external_packages": 2,
        "local_files": 2,
        "stdlib_modules": 11
    }
}

Class-Based Interface

For advanced use cases or multiple analyses:

from mlflow_dep_analyzer import UnifiedDependencyAnalyzer

# Create analyzer instance
analyzer = UnifiedDependencyAnalyzer(repo_root=".")

# Analyze multiple entry points
result = analyzer.analyze_dependencies(["model.py", "train.py", "utils.py"])

Convenience Functions

from mlflow_dep_analyzer import get_model_requirements, get_model_code_paths

# Get just the requirements list
packages = get_model_requirements("model.py")
# Returns: ["pandas", "scikit-learn", "numpy"]

# Get just the code paths list
files = get_model_code_paths("model.py")
# Returns: ["model.py", "utils.py", "preprocessing.py"]

🏗️ Architecture

The library uses a single, unified analyzer that provides complete dependency analysis:

┌─────────────────────────────┐
│   UnifiedDependencyAnalyzer │
│    (Complete Analysis)      │
└─────────────────────────────┘
                │
                ├─── AST parsing (safe import discovery)
                ├─── importlib.import_module() (dynamic imports)
                ├─── inspect.getsourcefile() (accurate file paths)
                ├─── Smart classification:
                │    ├─── Standard library → ignored
                │    ├─── External packages → requirements
                │    └─── Local files → code_paths + recursive analysis
                └─── MLflow-compatible output

🔍 How It Works

1. AST Parsing: Safely extracts import statements without executing code
2. Module Resolution: Uses importlib.import_module() + inspect.getsourcefile()
3. Smart Classification: Automatically categorizes modules:
   • 📦 External packages → Added to requirements
   • 🐍 Standard library → Ignored (built into Python)
   • 📁 Local files → Added to code_paths and analyzed recursively
4. Dependency Discovery: Recursively follows imports to build complete dependency graph
5. Path Optimization: Generates minimal file lists and package requirements

🌟 Advanced Usage

Complex Project Structure

from mlflow_dep_analyzer import UnifiedDependencyAnalyzer

# Analyze a complex project with src/ structure
analyzer = UnifiedDependencyAnalyzer(repo_root="/path/to/project")
result = analyzer.analyze_dependencies([
    "src/models/classifier.py",
    "src/models/preprocessor.py",
    "src/utils/data_loader.py"
])

print(f"Found {result['analysis']['total_modules']} total modules")
print(f"External packages: {result['analysis']['external_packages']}")
print(f"Local files: {result['analysis']['local_files']}")

Advanced Analysis

from mlflow_dep_analyzer import UnifiedDependencyAnalyzer

# Get detailed analysis results
analyzer = UnifiedDependencyAnalyzer(repo_root=".")
result = analyzer.analyze_dependencies(["model.py"])

# Access detailed metrics
print(f"Total modules found: {result['analysis']['total_modules']}")
print(f"External packages: {result['analysis']['external_packages']}")
print(f"Local files: {result['analysis']['local_files']}")
print(f"Standard library modules: {result['analysis']['stdlib_modules']}")

Error Handling

from mlflow_dep_analyzer import analyze_model_dependencies

try:
    result = analyze_model_dependencies("model.py")
except FileNotFoundError:
    print("Model file not found")
except ImportError as e:
    print(f"Import resolution failed: {e}")

🧪 Examples

See the examples/ directory for complete working examples:

• Basic Usage: Complete MLflow integration demo
• MLflow Integration: Real-world MLflow projects
• Complex Projects: Multi-file analysis with auto-logging

🛠️ Development

Setup

This project uses uv for dependency management:

git clone https://github.com/andrewgross/mlflow-dep-analyzer
cd mlflow-dep-analyzer
uv sync

Running Tests

# Run all tests
uv run pytest

# Run with coverage
uv run pytest --cov=src/mlflow_dep_analyzer --cov-report=html

# Run specific test categories
uv run pytest tests/test_unified_analyzer.py -v

Code Quality

# Linting and formatting
uv run ruff check
uv run ruff format

# Type checking
uv run mypy src/

# Pre-commit hooks
uv run pre-commit run --all-files

Requirements

• Python: 3.8+ (developed with 3.11.11 for Databricks Runtime 15.4 LTS compatibility)
• Core dependencies: MLflow 2.0+
• Development: pytest, ruff, mypy, pre-commit

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

Quick Contribution Guide

1. Fork the repository
2. Create a feature branch: git checkout -b feature-name
3. Make your changes with tests
4. Run the test suite: uv run pytest
5. Submit a pull request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Built on MLflow's production-tested dependency resolution utilities
• Inspired by the need for reliable, minimal MLflow model deployments
• Thanks to the Python AST and importlib developers for robust introspection tools

📈 Roadmap

• Configuration file support
• Plugin system for custom analyzers
• Integration with other ML frameworks
• Dependency vulnerability scanning
• Performance optimizations with caching

---

Documentation • Issues • Contributing

Made with ❤️ for the MLflow community