corp-extractor 0.2.2

Extract structured statements from text using T5-Gemma 2 and Diverse Beam Search

Corp Extractor

Extract structured subject-predicate-object statements from unstructured text using the T5-Gemma 2 model.

Features

• Structured Extraction: Converts unstructured text into subject-predicate-object triples
• Entity Type Recognition: Identifies 12 entity types (ORG, PERSON, GPE, LOC, PRODUCT, EVENT, etc.)
• Quality Scoring (v0.2.0): Each triple scored for groundedness (0-1) based on source text
• Beam Merging (v0.2.0): Combines top beams for better coverage instead of picking one
• Embedding-based Dedup (v0.2.0): Uses semantic similarity to detect near-duplicate predicates
• Predicate Taxonomies (v0.2.0): Normalize predicates to canonical forms via embeddings
• Contextualized Matching (v0.2.2): Compares full "Subject Predicate Object" against source text for better accuracy
• Multiple Output Formats: Get results as Pydantic models, JSON, XML, or dictionaries

Installation

# Recommended: include embedding support for smart deduplication
pip install corp-extractor[embeddings]

# Minimal installation (no embedding features)
pip install corp-extractor

Note: For GPU support, install PyTorch with CUDA first:

pip install torch --index-url https://download.pytorch.org/whl/cu121
pip install corp-extractor[embeddings]

Quick Start

from statement_extractor import extract_statements

result = extract_statements("""
    Apple Inc. announced the iPhone 15 at their September event.
    Tim Cook presented the new features to customers worldwide.
""")

for stmt in result:
    print(f"{stmt.subject.text} ({stmt.subject.type})")
    print(f"  --[{stmt.predicate}]--> {stmt.object.text}")
    print(f"  Confidence: {stmt.confidence_score:.2f}")  # NEW in v0.2.0

New in v0.2.0: Quality Scoring & Beam Merging

By default, the library now:

• Scores each triple for groundedness based on whether entities appear in source text
• Merges top beams instead of selecting one, improving coverage
• Uses embeddings to detect semantically similar predicates ("bought" ≈ "acquired")

from statement_extractor import ExtractionOptions, ScoringConfig

# Precision mode - filter low-confidence triples
scoring = ScoringConfig(min_confidence=0.7)
options = ExtractionOptions(scoring_config=scoring)
result = extract_statements(text, options)

# Access confidence scores
for stmt in result:
    print(f"{stmt} (confidence: {stmt.confidence_score:.2f})")

New in v0.2.0: Predicate Taxonomies

Normalize predicates to canonical forms using embedding similarity:

from statement_extractor import PredicateTaxonomy, ExtractionOptions

taxonomy = PredicateTaxonomy(predicates=[
    "acquired", "founded", "works_for", "announced",
    "invested_in", "partnered_with"
])

options = ExtractionOptions(predicate_taxonomy=taxonomy)
result = extract_statements(text, options)

# "bought" -> "acquired" via embedding similarity
for stmt in result:
    if stmt.canonical_predicate:
        print(f"{stmt.predicate} -> {stmt.canonical_predicate}")

New in v0.2.2: Contextualized Matching

Predicate canonicalization and deduplication now use contextualized matching:

• Compares full "Subject Predicate Object" strings against source text
• Better accuracy because predicates are evaluated in context
• When duplicates are found, keeps the statement with the best match to source text

This means "Apple bought Beats" vs "Apple acquired Beats" are compared holistically, not just "bought" vs "acquired".

Disable Embeddings (Faster, No Extra Dependencies)

options = ExtractionOptions(
    embedding_dedup=False,  # Use exact text matching
    merge_beams=False,      # Select single best beam
)
result = extract_statements(text, options)

Output Formats

from statement_extractor import (
    extract_statements,
    extract_statements_as_json,
    extract_statements_as_xml,
    extract_statements_as_dict,
)

# Pydantic models (default)
result = extract_statements(text)

# JSON string
json_output = extract_statements_as_json(text)

# Raw XML (model's native format)
xml_output = extract_statements_as_xml(text)

# Python dictionary
dict_output = extract_statements_as_dict(text)

Batch Processing

from statement_extractor import StatementExtractor

extractor = StatementExtractor(device="cuda")  # or "cpu"

texts = ["Text 1...", "Text 2...", "Text 3..."]
for text in texts:
    result = extractor.extract(text)
    print(f"Found {len(result)} statements")

Entity Types

Type	Description	Example
ORG	Organizations	Apple Inc., United Nations
PERSON	People	Tim Cook, Elon Musk
GPE	Geopolitical entities	USA, California, Paris
LOC	Non-GPE locations	Mount Everest, Pacific Ocean
PRODUCT	Products	iPhone, Model S
EVENT	Events	World Cup, CES 2024
WORK_OF_ART	Creative works	Mona Lisa, Game of Thrones
LAW	Legal documents	GDPR, Clean Air Act
DATE	Dates	2024, January 15
MONEY	Monetary values	$50 million, €100
PERCENT	Percentages	25%, 0.5%
QUANTITY	Quantities	500 employees, 1.5 tons
UNKNOWN	Unrecognized	(fallback)

How It Works

This library uses the T5-Gemma 2 statement extraction model with Diverse Beam Search (Vijayakumar et al., 2016):

1. Diverse Beam Search: Generates 4+ candidate outputs using beam groups with diversity penalty
2. Quality Scoring (v0.2.0): Each triple scored for groundedness in source text
3. Beam Merging (v0.2.0): Top beams combined for better coverage
4. Embedding Dedup (v0.2.0): Semantic similarity removes near-duplicate predicates
5. Predicate Normalization (v0.2.0): Optional taxonomy matching via embeddings
6. Contextualized Matching (v0.2.2): Full statement context used for canonicalization and dedup

Requirements

• Python 3.10+
• PyTorch 2.0+
• Transformers 4.35+
• Pydantic 2.0+
• sentence-transformers 2.2+ (optional, for embedding features)
• ~2GB VRAM (GPU) or ~4GB RAM (CPU)

Links

• Model on HuggingFace
• Web Demo
• Diverse Beam Search Paper
• Corp-o-Rate

License

MIT License - see LICENSE file for details.