Extracta

Deprecated. This package is no longer maintained. For document text extraction use markitdown directly. Existing installs continue to work but no further updates will be published.

Modular Content Analysis Platform for research, assessment, and academic integrity checking.

Extracta provides a unified interface for extracting and analyzing content from diverse media types including documents, images, repositories, and web content. It supports both research-focused deep analysis and assessment-oriented quality evaluation, with specialized tools for academic integrity validation.

✨ Key Features

• 🧩 Modular Architecture: Pluggable lenses and analyzers for different content types
• 📚 Academic Integrity: Citation-reference validation, bibliography checking, URL verification, AI conversation analysis
• 🤖 AI Conversation Analysis: Cognitive intent classification for AI-assisted learning assessment
• 🔍 Multiple Analysis Modes: Research and assessment workflows
• 📄 Rich Content Support: Text, images, documents, repositories, presentations, spreadsheets, AI conversations
• 🎯 Rubric-Based Assessment: Custom rubrics for structured evaluation
• 🛡️ Security First: Input sanitization, URL validation, malicious content detection
• 🧠 Intelligent Analysis: Pattern detection, quality scoring, integrity validation, learning pattern recognition
• 💻 Multiple Interfaces: CLI, Python API, and Web API
• 🔧 Modern Python: Built with uv, ruff, mypy, and pytest

Installation

From PyPI

pip install extracta

From Source

git clone https://github.com/michaelborck-education/extracta.git
cd extracta
pip install -e .

Optional Dependencies

Install with specific feature support:

pip install extracta[audio]     # Audio processing (faster-whisper for Apple Silicon)
pip install extracta[video]     # Video processing
pip install extracta[text]      # Enhanced text analysis (spaCy, NLTK)
pip install extracta[image]     # Image analysis with OCR
pip install extracta[code]      # Code analysis
pip install extracta[citation]  # Academic integrity (CrossRef, URL validation)
pip install extracta[conversation]  # AI conversation analysis (Gemini default)
pip install extracta[openai]    # OpenAI LLM provider
pip install extracta[claude]    # Anthropic Claude LLM provider
pip install extracta[openrouter] # OpenRouter unified API
pip install extracta[api]       # Web API server (FastAPI, Uvicorn)
pip install extracta[all]       # All features

Usage

Command Line

Basic Content Analysis

# Analyze document for research insights
extracta analyze research_paper.pdf --mode research --output analysis.json

# Assess student submission quality
extracta analyze essay.docx --mode assessment --output feedback.json

# Analyze repository structure and content
extracta analyze https://github.com/user/repo --mode assessment

Academic Integrity Checking

# Comprehensive citation and reference validation
extracta citation analyze student_paper.pdf --output integrity_check.json

# AI conversation cognitive intent analysis (with different LLM providers)
extracta citation conversation chatgpt_export.json --provider gemini --output analysis.json
extracta citation conversation chat.json --provider claude --model claude-3-sonnet-20240229
extracta citation conversation chat.json --provider openai --model gpt-4
extracta citation conversation chat.json --provider openrouter --model anthropic/claude-3-haiku

# Results include:
# - Citation-reference relationship validation
# - Bibliography padding detection
# - URL accessibility and domain reputation
# - AI conversation learning pattern analysis
# - Academic integrity scoring

Python API

Basic Content Analysis

from extracta import TextAnalyzer

analyzer = TextAnalyzer()
result = analyzer.analyze(text_content, mode="research")
print(result)

Academic Integrity Analysis

from extracta.analyzers import CitationAnalyzer, ReferenceAnalyzer, URLAnalyzer, ConversationAnalyzer

# Citation-reference validation
citation_analyzer = CitationAnalyzer()
citation_result = citation_analyzer.analyze(document_text)

# Bibliography quality assessment
reference_analyzer = ReferenceAnalyzer()
reference_result = reference_analyzer.analyze(document_text)

# URL validation and reputation checking
url_analyzer = URLAnalyzer()
url_result = url_analyzer.analyze(document_text)

# AI conversation cognitive intent analysis (with different providers)
conversation_analyzer = ConversationAnalyzer(provider="claude", model="claude-3-sonnet-20240229")
conversation_result = conversation_analyzer.analyze(conversation_json_data)

# Or use OpenAI
conversation_analyzer = ConversationAnalyzer(provider="openai", model="gpt-4")
conversation_result = conversation_analyzer.analyze(conversation_json_data)

# Combined integrity score
integrity_score = citation_result['citation_analysis']['academic_integrity_score']
learning_quality = conversation_result['conversation_analysis']['learning_assessment']['learning_quality_score']
print(f"Academic Integrity Score: {integrity_score}/100")
print(f"AI Learning Quality Score: {learning_quality}/100")

Grading and Assessment

from extracta.grading.rubric_manager import RubricRepository, get_default_rubric
from extracta.grading.feedback_generator import FeedbackGenerator

# Load or create a rubric
repo = RubricRepository("rubrics")
rubric = get_default_rubric("academic")  # or repo.load("my-rubric")

# Generate feedback based on analysis results
generator = FeedbackGenerator()
feedback = generator.generate_feedback(
    rubric=rubric,
    analysis_data=analysis_result,
    audience="student",
    detail="detailed"
)

🎓 Academic Integrity Features

Extracta provides comprehensive tools for detecting academic integrity issues and validating scholarly work:

Citation Analysis

• Citation-Reference Validation: Ensures all references have corresponding in-text citations
• Bibliography Padding Detection: Identifies references without citations
• Citation Stuffing Detection: Flags excessive citations in single sentences
• Style Recognition: Supports APA, MLA, Chicago, Harvard, and Numeric styles

Reference Validation

• DOI Verification: Validates Digital Object Identifiers with CrossRef API
• URL Accessibility: Checks if referenced URLs are accessible (404 detection)
• Domain Reputation: Analyzes source credibility (academic vs. commercial domains)
• Format Validation: Ensures proper reference formatting and completeness

AI Conversation Analysis

• Cognitive Intent Classification: Uses configurable LLM to classify user prompts as Delegation vs. Scaffolding
• Multi-Provider Support: Gemini, OpenAI GPT, Anthropic Claude, OpenRouter unified API
• Learning Pattern Recognition: Analyzes conversation flow for active learning behaviors
• Session Quality Scoring: Provides learning quality assessment (0-100)
• Platform Support: ChatGPT, Claude, Bard, and generic conversation formats

Security & Privacy

• Input Sanitization: Detects and prevents malicious content, hidden text, and LLM jailbreaks
• URL Protection: SSRF prevention with academic domain whitelisting
• Content Validation: Size limits, encoding validation, and integrity checking
• Privacy First: No data persistence, user-controlled processing, ephemeral analysis
• Safe Processing: Static analysis only, no code execution or external script running

Repository Analysis

• WordPress Detection: Identifies WordPress projects and analyzes themes/plugins
• Code Quality Assessment: Evaluates repository structure and practices
• File Type Analysis: Comprehensive analysis of all repository contents

Integrity Scoring

• Academic Integrity Score: 0-100 scale based on multiple validation criteria
• Detailed Reporting: Specific issues and recommendations
• Pattern Detection: Identifies suspicious citation and reference patterns

Development

Setup

# Clone repository
git clone https://github.com/michaelborck-education/extracta.git
cd extracta

# Create virtual environment
uv venv
source .venv/bin/activate

# Install dependencies
uv pip install -e ".[dev]"

Testing

# Run tests
pytest

# With coverage
pytest --cov=extracta

Linting and Type Checking

# Lint with ruff
ruff check .

# Type check with mypy
mypy extracta

# Format code
ruff format .

Building and Publishing

# Build package
uv build

# Publish to PyPI
uv venv  # if not already
source .venv/bin/activate
uv pip install twine
twine upload dist/* --repository pypi

Project Structure

extracta/
├── extracta/
│   ├── lenses/              # Content extraction modules
│   │   ├── audio_lens/      # Audio file processing
│   │   ├── video_lens/      # Video file processing
│   │   ├── image_lens/      # Image processing with OCR
│   │   ├── document_lens/   # Text & Office document processing
│   │   ├── presentation_lens/ # Presentation file analysis
│   │   ├── repo_lens/       # Repository-level analysis
│   │   └── base_lens.py     # Common lens interface
│   ├── analyzers/           # Content analysis modules
│   │   ├── text_analyzer/   # Text quality and readability
│   │   ├── image_analyzer/  # Image quality assessment
│   │   ├── citation_analyzer/ # Citation-reference validation
│   │   ├── reference_analyzer/ # Bibliography quality assessment
│   │   ├── url_analyzer/    # URL validation and reputation
│   │   └── base_analyzer.py # Common analyzer interface
│   ├── grading/             # Assessment and grading
│   │   ├── rubric_manager/  # Rubric creation and management
│   │   └── feedback_generator.py # AI-powered feedback
│   ├── orchestration/       # Workflow management
│   ├── shared/              # Common utilities
│   └── cli/                 # Command-line interface
├── tests/                   # Test suite
├── docs/                    # Documentation
├── examples/                # Usage examples
├── pyproject.toml           # Package configuration
└── README.md               # This file

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Run the test suite
6. Submit a pull request

License

MIT License - see LICENSE file for details.

🚀 Current Status & Roadmap

✅ Implemented Features

• Text Analysis: Readability, sentiment, vocabulary, quality metrics
• Image Analysis: OCR, quality assessment, accessibility
• Document Processing: PDF, DOCX, Office docs (PPTX, Excel, CSV)
• Citation Validation: Citation-reference relationships, academic integrity
• Reference Analysis: Bibliography quality, DOI validation, CrossRef integration
• URL Validation: Accessibility checking, domain reputation, robots.txt
• AI Conversation Analysis: Cognitive intent classification, learning pattern recognition
• Repository Analysis: GitHub repo analysis, WordPress detection
• Rubric System: Custom rubrics, structured assessment
• CLI Interface: Multiple commands for different analysis types
• Web API: REST API for integration
• Python API: Programmatic access

🔄 In Development

• Audio Lens: Speech-to-text, audio quality analysis
• Video Lens: Frame analysis, transcript processing
• Code Analyzer: Code quality metrics, best practices
• Screenshot Integration: Visual URL validation
• Wayback Machine: Archive URL checking

📋 Future Enhancements

• URL Conversation Input: Direct analysis of conversations from URLs (ChatGPT share links, etc.)
• GUI Application: Web-based interface
• LMS Integration: Canvas, Blackboard, Moodle
• Advanced ML Models: Fine-tuned for educational content
• Collaborative Features: Multi-user assessment workflows
• Plugin Architecture: Custom lenses and analyzers