Data Extraction with Language Models

DELM is a Python toolkit for extracting structured data from unstructured text using language models.

📖 Full Documentation

Features

• Multiple input formats: TXT, HTML, MD, DOCX, PDF, CSV, Excel, Parquet, Feather
• Flexible schemas: Simple key-value → nested objects → multiple schemas
• Multiple LLM providers: OpenAI, Anthropic, Google, Groq, Together AI, Fireworks AI
• Cost management: Automatic cost tracking, caching, and budget limits
• Built for scale: Batch processing with parallel execution and checkpointing

Installation

pip install delm

Quick Start

Define your extraction schema and extract structured data in just a few lines:

from delm import DELM, Schema, ExtractionVariable

# Define what to extract
schema = Schema.simple(
    variables_list=[
        ExtractionVariable(
            name="company",
            description="Company name mentioned",
            data_type="string",
            required=True,
        ),
        ExtractionVariable(
            name="price",
            description="Price value if mentioned",
            data_type="number",
            required=False,
        ),
    ]
)

# Initialize and extract
delm = DELM(
    schema=schema,
    provider="openai",
    model="gpt-4o-mini",
)

# Extract from any supported file format
results = delm.extract("data/earnings_calls.txt")
print(results)

# Check costs
print(delm.get_cost_summary())

Schema Types

DELM supports three schema types for different extraction needs:

Simple Schema

Extract key-value pairs from text:

schema = Schema.simple(
    variables_list=[
        ExtractionVariable(name="author", data_type="string"),
        ExtractionVariable(name="date", data_type="date"),
    ]
)

Nested Schema

Extract lists of structured objects:

schema = Schema.nested(
    container_name="products",
    variables_list=[
        ExtractionVariable(name="name", data_type="string"),
        ExtractionVariable(name="price", data_type="number"),
        ExtractionVariable(name="features", data_type="[string]"),
    ]
)

Multiple Schemas

Extract multiple different schemas simultaneously:

schema = Schema.multiple({
    "companies": Schema.nested(
        container_name="companies",
        variables_list=[...],
    ),
    "products": Schema.nested(
        container_name="products",
        variables_list=[...],
    ),
})

Supported Data Types

Advanced Features

Custom Prompts

delm = DELM(
    schema=schema,
    provider="openai",
    model="gpt-4o-mini",
    prompt_template="""You are a financial data extraction expert.

Extract the following information:
{variables}

Text to analyze:
{text}""",
)

Process CSV/Structured Data

delm = DELM(
    schema=schema,
    provider="openai",
    model="gpt-4o-mini",
    target_column="transcript_text",  # Column containing text to process
)

results = delm.extract("earnings_data.csv")

Cost Tracking & Limits

delm = DELM(
    schema=schema,
    provider="openai",
    model="gpt-4o-mini",
    track_cost=True,
    max_budget=10.0,  # Stop if cost exceeds $10
)

results = delm.extract("data.txt")
summary = delm.get_cost_summary()
print(f"Total cost: ${summary['total_cost']:.2f}")

Batch Processing

delm = DELM(
    schema=schema,
    provider="openai",
    model="gpt-4o-mini",
    batch_size=50,      # Process 50 records per batch
    max_workers=5,      # Use 5 parallel workers
)

results = delm.extract("large_dataset.csv")

Configuration Options

For a complete list of configuration options, see the documentation.

Common parameters:

• provider: LLM provider ("openai", "anthropic", "google", etc.)
• model: Model name ("gpt-4o-mini", "claude-3-sonnet-20240229", etc.)
• temperature: Generation temperature (default: 0.0)
• batch_size: Records per batch (default: 10)
• max_workers: Concurrent workers (default: 1)
• track_cost: Enable cost tracking (default: True)
• max_budget: Maximum cost limit in dollars (default: None)
• target_column: Column name for CSV/tabular data (default: None)

Documentation

📖 Full Documentation

Learn more about:

• Getting Started Guide
• Schema Design
• Text Processing & Filtering
• Cost Management
• API Reference

File Format Support

Contributing

We welcome contributions! Please see our documentation for guidelines.

License

This project is licensed under the MIT License - see the LICENSE.md file for details.

Acknowledgments

• Built on Instructor for structured outputs
• Uses Marker for PDF processing
• Developed at the Center for Applied AI at Chicago Booth