metaminer 0.3.4

Extract structured information from documents using AI

Metaminer

A tool for extracting structured information from documents using AI.

Overview

Metaminer allows you to extract structured data from various document formats (PDF, DOCX, TXT, etc.) by asking natural language questions. It uses AI to analyze documents and return structured results in CSV or JSON format.

Installation

pip install metaminer

System Requirements

Metaminer requires pandoc to be installed on your system for document processing:

• Ubuntu/Debian: sudo apt-get install pandoc
• macOS: brew install pandoc
• Windows: Download from pandoc.org

Note: Metaminer uses pandoc for most document formats and PyMuPDF specifically for PDF processing to ensure optimal text extraction.

Usage

Command Line Interface

# Basic usage
metaminer questions.txt documents/

# Process single document
metaminer questions.txt document.pdf

# Save results to file
metaminer questions.txt documents/ --output results.csv

# JSON output format
metaminer questions.txt documents/ --format json --output results.json

# Custom API endpoint
metaminer questions.txt documents/ --base-url http://localhost:8000/api/v1

# Show normalized question structure with inferred types
metaminer questions.txt --show-questions --output questions_analysis.csv

# Verbose output for debugging
metaminer questions.txt documents/ --verbose

# Custom API key
metaminer questions.txt documents/ --api-key your-api-key

Python Module

from metaminer import Inquiry, extract_metadata, Config
from metaminer import extract_text, extract_text_from_directory, get_supported_extensions
from metaminer import DataTypeInferrer, infer_question_types, setup_logging
import pandas as pd

# From question file
inquiry = Inquiry.from_file("questions.txt")
df = inquiry.process_documents("documents/")

# Direct questions
inquiry = Inquiry(questions=["Who is the author?", "What is the publication date?"])
df = inquiry.process_documents(["doc1.pdf", "doc2.docx"])

# Single document
result = inquiry.process_document("document.pdf")

# Extract text directly
text = extract_text("document.pdf")

# Extract text from directory
texts = extract_text_from_directory("documents/")

# Get supported file extensions
extensions = get_supported_extensions()

# Use configuration
config = Config()
print(f"Default API endpoint: {config.base_url}")

# Set up logging
logger = setup_logging(config)

# Infer data types for questions
questions = ["Who is the author?", "What is the publication date?", "How many pages?"]
type_suggestions = infer_question_types(questions)
for q, suggestion in type_suggestions.items():
    print(f"{q}: {suggestion.suggested_type} - {suggestion.reasoning}")

# Use DataTypeInferrer directly
inferrer = DataTypeInferrer()
suggestion = inferrer.infer_single_type("What is the priority level?")
print(f"Suggested type: {suggestion.suggested_type}")

Question Formats

Text File (.txt)

One question per line:

Who is the author?
What is the publication date?
What is the main topic?

CSV File (.csv)

Structured format with optional field names and data types:

question,field_name,data_type
"Who is the author?",author,str
"What is the publication date?",pub_date,date
"How many pages?",page_count,int

Supported data types:

• str (default): Text
• int: Integer numbers
• float: Decimal numbers
• bool: True/False values
• date: Date values (e.g., YYYY-MM-DD)
• datetime: Date and time values (e.g., YYYY-MM-DD HH:MM:SS)
• list(type): Arrays of values (e.g., list(str), list(int), list(date))
• enum(val1,val2,val3): Single choice from discrete values
• multi_enum(val1,val2,val3): Multiple choices from discrete values

Supported Document Formats

Thanks to pandoc integration and PyMuPDF, metaminer supports:

• PDF (.pdf)
• Microsoft Word (.docx, .doc)
• OpenDocument (.odt)
• Rich Text Format (.rtf)
• Plain text (.txt)
• Markdown (.md)
• HTML (.html)
• EPUB (.epub)
• LaTeX (.tex)

Configuration

API Settings

By default, metaminer connects to a local AI server at http://localhost:5001/api/v1. You can customize this using environment variables or command-line options:

Environment Variables

# API Configuration
export OPENAI_API_KEY=your-api-key
export METAMINER_BASE_URL=http://your-api-server.com/api/v1
export METAMINER_MODEL=gpt-4
export METAMINER_TIMEOUT=60
export METAMINER_MAX_RETRIES=5

# Logging Configuration
export METAMINER_LOG_LEVEL=DEBUG

Command Line

metaminer questions.txt documents/ --base-url http://your-api-server.com/api/v1

Python

from metaminer import Inquiry, Config

# Using configuration
config = Config()
inquiry = Inquiry.from_file("questions.txt", base_url="http://your-api-server.com/api/v1")

# Or set environment variables before creating Inquiry
import os
os.environ["METAMINER_BASE_URL"] = "http://your-api-server.com/api/v1"
inquiry = Inquiry.from_file("questions.txt")

Configuration Defaults

• Base URL: http://localhost:5001/api/v1
• Model: gpt-3.5-turbo
• Timeout: 30 seconds
• Max Retries: 3
• Log Level: INFO
• Max File Size: 50MB

Output Format

Results include the extracted information plus metadata:

author,pub_date,page_count,_document_path,_document_name
"John Doe","2023-01-15",25,"/path/to/doc1.pdf","doc1.pdf"
"Jane Smith","2023-02-20",18,"/path/to/doc2.pdf","doc2.pdf"

Examples

Research Paper Analysis

# questions.txt
Who are the authors?
What is the title?
What journal was this published in?
What is the publication year?
What is the main research question?
What methodology was used?

Invoice Processing

question,field_name,data_type
"What is the invoice number?",invoice_number,str
"What is the total amount?",total_amount,float
"What is the invoice date?",invoice_date,date
"Who is the vendor?",vendor_name,str
"What is the due date?",due_date,date

Legal Document Review

What type of document is this?
Who are the parties involved?
What is the effective date?
What is the termination date?
What are the key obligations?

Document Classification with Enums

question,field_name,data_type
"What is the document type?",doc_type,"enum(report,memo,letter,invoice)"
"What topics are covered?",topics,"multi_enum(finance,hr,marketing,operations)"
"What is the priority level?",priority,"enum(low,medium,high,urgent)"
"What is the title?",title,str
"Who is the author?",author,str

Note: When using enum types in CSV files, make sure to quote the entire type specification to prevent CSV parsing issues with commas.

Data Type Inference

Metaminer includes an intelligent data type inference system that can automatically suggest appropriate data types for your questions. This feature uses AI to analyze question content and recommend the most suitable data types.

Using Type Inference

from metaminer import DataTypeInferrer, infer_question_types

# Infer types for multiple questions
questions = [
    "Who is the author?",
    "What is the publication date?", 
    "How many pages are there?",
    "Is this document confidential?",
    "What is the priority level?"
]

type_suggestions = infer_question_types(questions)
for question_id, suggestion in type_suggestions.items():
    print(f"Question: {questions[int(question_id.split('_')[1])-1]}")
    print(f"Suggested type: {suggestion.suggested_type}")
    print(f"Reasoning: {suggestion.reasoning}")
    print(f"Alternatives: {suggestion.alternatives}")
    print()

CLI Type Analysis

You can also analyze your questions from the command line:

# Analyze questions and show suggested types
metaminer questions.txt --show-questions

# Save analysis to file
metaminer questions.txt --show-questions --output question_analysis.csv

This will output a structured analysis showing:

• Original questions
• Suggested data types
• Field names
• Reasoning for type suggestions

Type Inference Features

• Smart Analysis: Uses AI to understand question context and intent
• Fallback Logic: Provides sensible defaults when AI analysis fails
• Validation: Ensures all suggested types are valid metaminer data types
• Multiple Suggestions: Provides alternative type options
• Reasoning: Explains why each type was suggested

Development

Running Tests

pip install -e ".[dev]"
pytest

Project Structure

metaminer/
├── __init__.py          # Main exports
├── inquiry.py           # Core Inquiry class
├── document_reader.py   # Document text extraction
├── question_parser.py   # Question file parsing
├── schema_builder.py    # Pydantic schema generation
├── datatype_inferrer.py # AI-powered data type inference
├── extractor.py         # Metadata extraction utilities
├── config.py           # Configuration management
├── cli.py              # Command-line interface
└── __main__.py         # Module entry point

License

GNU Lesser General Public License v3.0 - see LICENSE file for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.