ADFMentor 1.0.0

Evaluate Azure Data Factory submissions from JSON/ZIP/folder inputs with comprehensive AI-driven grading and professional reporting.

ADFMentor

A Python package for evaluating Azure Data Factory (ADF) submissions with comprehensive AI-driven grading.

Provides accurate, fair assessment of ADF implementations including architecture, design quality, error handling, and best practices.

Features

• AI-Driven Grading (Default): Comprehensive quality evaluation covering architecture, design, error handling, parameterization, and best practices
• ADF JSON validation for ARM templates and repo-export style files
• Safe ZIP extraction with size and file-count limits
• Recursive discovery of ADF JSON artifacts
• Optional rule-based grading for fast component checking (backward compatible)
• Clear feedback with specific recommendations
• Secure API key management via environment variables

Installation

pip install ADFMentor

Quick Start

from ADFMentor import ADFMentor

# Initialize mentor
mentor = ADFMentor(api_key="your-api-key", model_name="gemini-2.0-flash-exp")

# Evaluate with comprehensive AI grading (default - most accurate and fair)
result = mentor.evaluate_adf(
    submission_path="path/to/submission.zip",
    question="Grade this ADF assignment"
)

print(f"Score: {result['score']}/100")
print(f"Feedback:\n{result['feedback']}")
print(f"Metadata:\n{result['metadata_report']}")

# Optional: Use custom grading rubric
custom_rubric = """
Evaluate focus on: error handling, parameterization, 
security practices, and performance optimization.
Score 0-100 with detailed feedback.
"""

result_custom = mentor.evaluate_adf(
    submission_path="path/to/submission.zip",
    prompt=custom_rubric,
    question="Assess ADF quality"
)

# Optional: Use fast rule-based grading (backward compatible)
result_fast = mentor.evaluate_adf(
    submission_path="path/to/submission.zip",
    use_rule_based=True
)

Environment Setup

1. Get a free Google API key: https://makersuite.google.com/app/apikey
2. Copy .env.example to .env:

   cp .env.example .env
   

3. Edit .env and add your API key:

   GEMINI_API_KEY=your_actual_key_here
   

4. Important: Never commit .env to git (already protected in .gitignore)

Test files automatically load from .env.

Supported Formats

• .json - ARM templates or repo-export JSON files
• .zip - ADF export packages (may include .txt notes)
• folder path - ADF project directory
• .txt - Optional written explanations (AI grading only)

AI Grading Evaluation

The system uses Gemini AI (default: gemini-2.0-flash-exp) for comprehensive evaluation including:

• Architecture & Design: Pipeline organization, activity flow, component relationships
• Error Handling: Exception handling, retry logic, resilience patterns
• Parameterization: Runtime flexibility, hardcoded values, reusability
• Best Practices: Naming conventions, schema definitions, performance, security
• Completeness: All required components, connections, configurations
• Code Quality: JSON structure, documentation, clarity

Returns: Score (0-100) + detailed feedback with specific improvements.

Security Validation

Safe ZIP extraction with:

• 200 file limit
• 50MB total size limit
• 10MB per-file limit
• Path traversal protection

Custom Rubrics (PowerBI-Style)

For specialized evaluations, define custom prompts and pass them to evaluate_adf():

from ADFMentor import ADFMentor
import os

mentor = ADFMentor(api_key=os.getenv("GEMINI_API_KEY"))

# Define specialized prompts (like PowerBI DAX/Visual/Text pattern)
prompts = {
    "pipeline": """Evaluate pipeline logic correctness (40 pts), activity selection (30 pts), 
                   performance optimization (20 pts), and completeness (10 pts). 
                   Provide 50-80 word feedback.""",
    
    "architecture": """Evaluate architecture design (40 pts), error handling (30 pts), 
                        security practices (20 pts), and documentation (10 pts). 
                        Provide 50-80 word feedback.""",
    
    "configuration": """Evaluate configuration accuracy (40 pts), parameter usage (35 pts), 
                         security implementation (15 pts), and best practices (10 pts). 
                         Provide 50-80 word feedback."""
}

# Use specific prompt for different evaluation types
result_pipeline = mentor.evaluate_adf(
    submission_path="submission.zip",
    prompt=prompts["pipeline"],
    question="Evaluate the pipeline logic"
)

result_arch = mentor.evaluate_adf(
    submission_path="submission.zip",
    prompt=prompts["architecture"],
    question="Evaluate the architecture"
)

# Or use default comprehensive AI grading (no prompt needed)
result_default = mentor.evaluate_adf(submission_path="submission.zip")

Key Features:

• Define prompts in your test file (like PowerBI pattern)
• Pass different prompts for different evaluation focuses
• Default AI grading works without custom prompts
• 50-80 word concise feedback

ADFMentor/
  __init__.py
ADFMentorInternal/
  core.py
  models/
    gemini.py
  utils/
    adf_validator.py
    adf_processor.py
    adf_evaluator.py

API

ADFMentor

• ADFMentor(api_key, model_name="gemini-2.0-flash-exp")
• evaluate_adf(submission_path, prompt=None, question=None)

Returns:

{
  "score": 0,
  "feedback": "...",
  "metadata_report": "..."
}

Notes:

• For .txt submissions, provide prompt (and optionally question) to enable AI grading.
• For .zip/folder submissions, .txt files are treated as optional companion notes in AI mode.

License

MIT