mark-mate 0.2.0

Your AI Teaching Assistant for Assignments and Assessment

MarkMate

Your AI Teaching Assistant for Assignments and Assessment

A comprehensive system for processing, consolidating, and grading student submissions with support for multiple content types, GitHub repository analysis, WordPress assignments, and AI-powered assessment using Claude 3.5 Sonnet, GPT-4o, and Google Gemini through a unified LiteLLM interface.

🚀 Quick Start

Installation

pip install mark-mate

GUI Application (Recommended)

Launch the cross-platform desktop application:

mark-mate-gui

The GUI provides an intuitive interface for all MarkMate functionality with:

• Visual workflow navigation between consolidate → scan → extract → grade
• Progress tracking for long-running operations
• Interactive configuration builder for grading settings
• Real-time status updates and error handling
• Cross-platform support (Windows, macOS, Linux)

CLI Usage

# Consolidate submissions
mark-mate consolidate raw_submissions/

# Scan for GitHub URLs
mark-mate scan processed_submissions/ --output github_urls.txt

# Extract content with analysis
mark-mate extract processed_submissions/ --github-urls github_urls.txt --output extracted.json

# Grade submissions (uses auto-configuration)
mark-mate grade extracted.json assignment_spec.txt --output results.json

# Use custom grading configuration
mark-mate grade extracted.json assignment_spec.txt --config grading_config.yaml

API Keys Setup

export ANTHROPIC_API_KEY="your_anthropic_key"
export OPENAI_API_KEY="your_openai_key"
export GEMINI_API_KEY="your_gemini_key"  # or GOOGLE_API_KEY

📋 System Overview

MarkMate consists of four main components accessible via GUI, CLI, or Python API:

1. Consolidate - File organization and filtering

• Groups files by student ID with intelligent pattern matching
• Extracts zip archives with conflict resolution
• Filters Mac system files (.DS_Store, resource forks, __MACOSX)
• WordPress mode for UpdraftPlus backup organization

2. Scan - GitHub repository detection

• Comprehensive URL detection with regex patterns
• Scans text files within zip archives
• Enhanced encoding support for international students
• Creates editable student_id:repo_url mapping files

3. Extract - Multi-format content processing

• Document Processing: PDF, DOCX, TXT, MD, Jupyter notebooks
• Code Analysis: Python, HTML, CSS, JavaScript, React/TypeScript
• GitHub Analysis: Commit history, development patterns, repository quality
• WordPress Processing: Themes, plugins, database, AI detection
• Enhanced Encoding: 18+ encodings for international students

4. Grade - AI-powered assessment

• Multi-LLM Grading: Claude 3.5 Sonnet + GPT-4o + Google Gemini via LiteLLM
• Enhanced Multi-Run System: Statistical averaging with configurable runs per grader
• Flexible Configuration: YAML-based grader setup with weights and priorities
• Advanced Statistics: Multiple averaging methods, confidence scoring, variance analysis
• Cost Controls: Budget limits, rate limiting, and usage tracking
• Automatic Rubric Extraction: From assignment specifications
• Comprehensive Feedback: Incorporating all analysis types

🌟 Key Features

• Cross-Platform Desktop GUI: Native desktop application powered by Flutter for Windows, macOS, and Linux
• Multi-Format Support: PDF, DOCX, TXT, MD, Jupyter notebooks, Python code, web files (HTML/CSS/JS), React/TypeScript projects
• GitHub Repository Analysis: Commit history, development patterns, repository quality assessment
• Enhanced Encoding Support: Optimized for ESL students with automatic encoding detection (UTF-8, UTF-16, CP1252, Latin-1, and more)
• WordPress Assignment Processing: Complete backup analysis with theme, plugin, and database evaluation
• Multi-LLM Grading: Claude 3.5 Sonnet + GPT-4o + Google Gemini with statistical aggregation and confidence scoring
• Mac System File Filtering: Automatic removal of .DS_Store, resource forks, and __MACOSX directories

🔗 LiteLLM Integration

MarkMate leverages LiteLLM for unified access to multiple LLM providers:

Unified Interface Benefits:

• Consistent API: Single interface for Claude, OpenAI, and Gemini
• Error Handling: Standardized exceptions across all providers
• Token Tracking: Unified usage monitoring and cost calculation
• Rate Limiting: Built-in respect for provider-specific limits
• Future-Proof: Easy addition of 100+ supported providers

Supported Models:

# Anthropic
"claude-3-5-sonnet", "claude-3-sonnet", "claude-3-haiku"

# OpenAI  
"gpt-4o", "gpt-4o-mini", "gpt-4", "gpt-3.5-turbo"

# Google Gemini
"gemini-pro", "gemini-1.5-pro", "gemini-2.0-flash"

Cost Optimization:

• Automatic cost estimation before processing
• Per-student budget controls with early termination
• Real-time usage tracking across all providers
• Intelligent model selection based on cost/quality trade-offs

🖥️ CLI Interface

Consolidate Command

mark-mate consolidate [OPTIONS] FOLDER_PATH

Options:
  --no-zip              Discard zip files instead of extracting
  --wordpress           Enable WordPress-specific processing
  --keep-mac-files      Preserve Mac system files
  --output-dir TEXT     Output directory (default: processed_submissions)

Scan Command

mark-mate scan [OPTIONS] SUBMISSIONS_FOLDER

Options:
  --output TEXT         Output file for URL mappings (default: github_urls.txt)
  --encoding TEXT       Text encoding (default: utf-8)

Extract Command

mark-mate extract [OPTIONS] SUBMISSIONS_FOLDER

Options:
  --output TEXT         Output JSON file (default: extracted_content.json)
  --wordpress           Enable WordPress processing
  --github-urls TEXT    GitHub URL mapping file
  --dry-run             Preview processing without extraction
  --max-students INT    Limit number of students (for testing)

Grade Command

mark-mate grade [OPTIONS] EXTRACTED_CONTENT ASSIGNMENT_SPEC

Options:
  --output TEXT         Output JSON file (default: grading_results.json)
  --rubric TEXT         Separate rubric file
  --max-students INT    Limit number of students
  --dry-run             Preview grading without API calls
  --config TEXT         Path to YAML grading configuration file (optional - uses defaults if not provided)

🐍 Python API

Library Usage

from mark_mate import (
    EnhancedGradingSystem, LLMProvider,
    AssignmentProcessor, ContentAnalyzer, GradingConfigManager
)

# Process submissions
processor = AssignmentProcessor()
result = processor.process_submission(
    "/path/to/submission", 
    "123", 
    wordpress=True,
    github_url="https://github.com/user/repo"
)

# Grading with custom configuration
grader = EnhancedGradingSystem("grading_config.yaml")
grade_result = grader.grade_submission(
    student_data=result,
    assignment_spec="Assignment requirements..."
)

# Grading with auto-configuration (no config file)
auto_grader = EnhancedGradingSystem()  # Uses defaults based on available API keys
auto_result = auto_grader.grade_submission(
    student_data=result,
    assignment_spec="Assignment requirements..."
)

# Direct LLM provider usage
llm_provider = LLMProvider()
llm_result = llm_provider.grade_submission(
    provider="gemini",
    model="gemini-1.5-pro", 
    prompt="Grade this submission..."
)

# Generate configuration programmatically
config_manager = GradingConfigManager()
config = config_manager.load_config()  # Load defaults
config.save_default_config("my_config.yaml")

# Analyze content
analyzer = ContentAnalyzer()
summary = analyzer.generate_submission_summary(
    result["content"], 
    result["metadata"]
)

🔄 Complete Workflows

Programming Assignment with GitHub

# 1. Consolidate submissions
mark-mate consolidate programming_submissions/

# 2. Scan for GitHub URLs
mark-mate scan processed_submissions/ --output github_urls.txt

# 3. Extract with comprehensive analysis
mark-mate extract processed_submissions/ --github-urls github_urls.txt

# 4. Grade with repository analysis
mark-mate grade extracted_content.json programming_assignment.txt

WordPress Assignment

# 1. Consolidate WordPress backups
mark-mate consolidate wordpress_submissions/ --wordpress

# 2. Extract WordPress content
mark-mate extract processed_submissions/ --wordpress

# 3. Grade with WordPress criteria
mark-mate grade extracted_content.json wordpress_assignment.txt

International Student Support

# Enhanced encoding detection handles international submissions automatically
mark-mate consolidate international_submissions/
mark-mate extract processed_submissions/  # Auto-detects 18+ encodings
mark-mate grade extracted_content.json assignment.txt

Advanced Grading Workflows

# Auto-configuration based on available API keys
mark-mate grade extracted_content.json assignment.txt

# Custom configuration with multiple providers and runs
mark-mate grade extracted_content.json assignment.txt --config grading_config.yaml

# Generate configuration templates
mark-mate generate-config --template minimal --output simple_config.yaml
mark-mate generate-config --template cost-optimized --output cheap_config.yaml

⚙️ Configuration Management

Automatic Configuration

MarkMate automatically creates optimal configurations based on your available API keys:

• Single Provider: Uses 3 runs for statistical reliability
• Multiple Providers: Uses 1 run each with weighted averaging
• Cost Optimization: Chooses appropriate models and limits

Configuration Templates

Generate ready-to-use configurations:

# Full-featured with all providers
mark-mate generate-config --template full

# Minimal single-provider setup  
mark-mate generate-config --template minimal

# Cost-optimized for budget constraints
mark-mate generate-config --template cost-optimized

# Single provider focus
mark-mate generate-config --template single-provider --provider anthropic

🎯 Enhanced Grading System

Multi-Run Statistical Grading

MarkMate's enhanced grading system provides unprecedented reliability through:

Statistical Aggregation Methods:

• Mean: Simple average of all runs
• Median: Middle value for robustness against outliers
• Weighted Mean: Importance-based averaging using grader weights
• Trimmed Mean: Removes highest/lowest values before averaging

Configuration Example:

grading:
  runs_per_grader: 3
  averaging_method: "weighted_mean"
  parallel_execution: true

graders:
  - name: "claude-sonnet"
    provider: "anthropic"
    model: "claude-3-5-sonnet"
    weight: 2.0              # Higher importance
    primary_feedback: true   # Featured feedback
    
  - name: "gpt4o"
    provider: "openai"
    model: "gpt-4o"
    weight: 1.5
    
  - name: "gemini-pro"
    provider: "gemini"
    model: "gemini-1.5-pro"
    weight: 1.0

execution:
  max_cost_per_student: 0.75  # Budget control
  retry_attempts: 3           # Failure handling

Advanced Features:

• Confidence Scoring: Based on inter-grader agreement and variance
• Cost Controls: Per-student budget limits and usage tracking
• Rate Limiting: Respects API limits for each provider
• Failure Recovery: Graceful degradation with retry logic
• Progress Tracking: Real-time status for large batches

Output Example:

{
  "aggregate": {
    "mark": 87.3,
    "feedback": "Excellent implementation with clear documentation...",
    "confidence": 0.92,
    "graders_used": 3,
    "total_runs": 9
  },
  "grader_results": {
    "claude-sonnet": {
      "aggregated": {"mark": 88.0, "runs_used": 3},
      "weight": 2.0
    }
  }
}

🌍 Enhanced Support for International Students

Advanced Encoding Detection

Comprehensive support for ESL (English as a Second Language) students:

Supported Encodings:

• UTF-16: Windows systems with non-English locales
• CP1252: Windows-1252 (Western European, legacy systems)
• Latin-1: ISO-8859-1 (European systems, older editors)
• Regional: Cyrillic (CP1251), Turkish (CP1254), Chinese (GB2312, Big5), Japanese (Shift_JIS), Korean (EUC-KR)

Intelligent Fallback Strategy:

1. Try optimal encoding based on content type
2. Graceful fallback with error handling
3. Preserve international characters and symbols
4. Detailed logging of encoding attempts

🐙 GitHub Repository Analysis

Comprehensive Development Assessment

Analyzes student GitHub repositories to evaluate development processes:

Repository Analysis Features:

• Commit History: Development timeline, frequency patterns, consistency
• Message Quality: Scoring based on descriptiveness and professionalism
• Development Patterns: Steady development vs. last-minute work detection
• Collaboration: Multi-author analysis, teamwork evaluation
• Repository Quality: README, documentation, directory structure
• Code Organization: File management, naming conventions, best practices

Analysis Output Example:

{
  "github_metrics": {
    "total_commits": 15,
    "development_span_days": 14,
    "commit_message_quality": {
      "score": 89,
      "quality_level": "excellent"
    },
    "consistency_score": 0.86,
    "collaboration_level": "collaborative"
  }
}

🎯 WordPress Assignment Support

Static Assessment Capabilities

Assess WordPress assignments without requiring site restoration:

Technical Implementation:

• Theme analysis and customization assessment
• Plugin inventory and functionality review
• Database content extraction and analysis
• Security configuration evaluation

Content Quality Assessment:

• Blog post count and word count analysis
• Media usage and organization
• User account configuration
• Comment analysis

AI Integration Detection:

• Automatic detection of AI-related plugins
• AI keyword analysis in plugin descriptions
• Assessment of AI integration documentation

📊 Output and Results

Extraction Output

{
  "extraction_session": {
    "timestamp": "2025-06-19T10:30:00",
    "total_students": 24,
    "wordpress_mode": true,
    "github_analysis": true
  },
  "students": {
    "123": {
      "content": {...},
      "metadata": {...}
    }
  }
}

Grading Output

{
  "grading_session": {
    "timestamp": "2025-06-19T11:00:00",
    "total_students": 24,
    "providers": ["claude", "openai"]
  },
  "results": {
    "123": {
      "aggregate": {
        "mark": 85,
        "feedback": "Comprehensive feedback...",
        "confidence": 0.95,
        "max_mark": 100
      },
      "providers": {
        "claude": {"mark": 83, "feedback": "..."},
        "openai": {"mark": 87, "feedback": "..."}
      }
    }
  }
}

🛠️ Development

Setup Development Environment

git clone https://github.com/markmate-ai/mark-mate.git
cd mark-mate
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -e ".[dev]"

Running Tests

pytest tests/

Code Quality

black src/
flake8 src/
mypy src/

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Adding New Features

1. Content Extractors: Follow the BaseExtractor pattern in src/mark_mate/extractors/
2. Analysis Capabilities: Extend existing analyzers or create new ones
3. LLM Providers: Add new providers in src/mark_mate/core/grader.py
4. CLI Commands: Add new commands in src/mark_mate/cli/

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🔗 Links

• Documentation: https://mark-mate.readthedocs.io
• GitHub: https://github.com/markmate-ai/mark-mate
• PyPI: https://pypi.org/project/mark-mate/
• Issues: https://github.com/markmate-ai/mark-mate/issues

🙏 Acknowledgments

MarkMate is designed for educational assessment purposes. Please ensure compliance with your institution's policies regarding automated grading and student data processing.

---

MarkMate: Your AI Teaching Assistant for Assignments and Assessment