Data Extraction with Language Models

DELM is a Python toolkit for extracting structured data from unstructured text using language models.

Load → split → score → filter → extract with schema validation → evaluate, all checkpointed so interrupted runs resume where they stopped.

📖 Full Documentation

Features

• Multiple input formats: TXT, HTML, MD, DOCX, PDF, CSV, Excel, Parquet, Feather
• Flexible schemas: Simple key-value → nested objects → multiple schemas
• Multiple LLM providers: OpenAI, Anthropic, Google, Groq, Together AI, Fireworks AI
• Cost management: Automatic cost tracking, caching, and budget limits
• Built for scale: Batch processing with parallel execution and checkpointing

Installation

pip install delm

Agent Skill

The LLM labeling skill guides coding agents to use DELM for labeling, annotation, classification, and structured extraction. Copy skills/llm-labeling/ into your agent's skills directory to use it.

Quick Start

Define your extraction schema and extract structured data in just a few lines:

from delm import DELM, Schema, ExtractionVariable

# Define what to extract
schema = Schema.simple(
    variables_list=[
        ExtractionVariable(
            name="company",
            description="Company name mentioned",
            data_type="string",
            required=True,
        ),
        ExtractionVariable(
            name="price",
            description="Price value if mentioned",
            data_type="number",
            required=False,
        ),
    ]
)

# Initialize and extract
delm = DELM(
    schema=schema,
    provider="openai",
    model="gpt-4o-mini",
)

# Extract from any supported file format
results = delm.extract("data/earnings_calls.txt")
print(results)

# Check costs
print(delm.get_cost_summary())

Schema Types

DELM supports three schema types for different extraction needs:

Simple Schema

Extract key-value pairs from text:

schema = Schema.simple(
    variables_list=[
        ExtractionVariable(name="author", data_type="string"),
        ExtractionVariable(name="date", data_type="date"),
    ]
)

Nested Schema

Extract lists of structured objects:

schema = Schema.nested(
    container_name="products",
    variables_list=[
        ExtractionVariable(name="name", data_type="string"),
        ExtractionVariable(name="price", data_type="number"),
        ExtractionVariable(name="features", data_type="[string]"),
    ]
)

Multiple Schemas

Extract multiple different schemas simultaneously:

schema = Schema.multiple({
    "companies": Schema.nested(
        container_name="companies",
        variables_list=[...],
    ),
    "products": Schema.nested(
        container_name="products",
        variables_list=[...],
    ),
})

Supported Data Types

Advanced Features

Custom Prompts

delm = DELM(
    schema=schema,
    provider="openai",
    model="gpt-4o-mini",
    prompt_template="""You are a financial data extraction expert.

Extract the following information:
{variables}

Text to analyze:
{text}""",
)

Process CSV/Structured Data

delm = DELM(
    schema=schema,
    provider="openai",
    model="gpt-4o-mini",
    target_column="transcript_text",  # Column containing text to process
)

results = delm.extract("earnings_data.csv")

Cost Tracking & Limits

delm = DELM(
    schema=schema,
    provider="openai",
    model="gpt-4o-mini",
    track_cost=True,
    max_budget=10.0,  # Stop if cost exceeds $10
)

results = delm.extract("data.txt")
summary = delm.get_cost_summary()
print(f"Total cost: ${summary['total_cost']:.2f}")

Batch Processing

delm = DELM(
    schema=schema,
    provider="openai",
    model="gpt-4o-mini",
    batch_size=50,      # Process 50 records per batch
    max_workers=5,      # Use 5 parallel workers
)

results = delm.extract("large_dataset.csv")

Configuration Options

For a complete list of configuration options, see the documentation.

Common parameters:

• provider: LLM provider ("openai", "anthropic", "google", etc.)
• model: Model name ("gpt-4o-mini", "claude-3-sonnet-20240229", etc.)
• temperature: Generation temperature (default: 0.0)
• batch_size: Records per batch (default: 10)
• max_workers: Concurrent workers (default: 1)
• track_cost: Enable cost tracking (default: True)
• max_budget: Maximum cost limit in dollars (default: None)
• target_column: Column name for CSV/tabular data (default: None)

Documentation

📖 Full Documentation

Learn more about:

• Getting Started Guide
• Schema Design
• Text Processing & Filtering
• Cost Management
• API Reference

File Format Support

Contributing

We welcome contributions! Please see our documentation for guidelines.

License

This project is licensed under the MIT License - see the LICENSE.md file for details.

Acknowledgments

• Built on Instructor for structured outputs
• Uses Marker for PDF processing
• Developed at the Center for Applied AI at Chicago Booth