creel 0.1.4

Extract linked information from a mess or sources

creel

Extract a typed graph from a mess of sources.

creel is a general, AI-powered source-to-graph extraction engine. You give it (a) sources — freeform prose, tables, JSON, PDFs; (b) a grammar of the graph you want — its node-types and edge-types and the typed values they carry; and (c) extractors — pluggable strategies that know how to find each element. creel returns a clean, auditable, typed property graph as a single source of truth, canonically a JSON graph specification:

extract(sources, graph_spec, extractors) -> graph

Everything downstream — persistence, query, graph-RAG, annotation, rendering to slides/reports — is a projection of that one graph.

Install

pip install creel                 # core: pydantic, jsonschema, networkx
pip install "creel[query]"        # SQL/JSON query extractors (duckdb, jmespath)
pip install "creel[ingest]"       # file loaders: PDF/DOCX/PPTX via docling, HTML via trafilatura, XLSX via openpyxl
pip install "creel[aix]"          # real LLM extraction/judging/embedding via aix (default)
pip install "creel[anthropic]"    # real LLM extraction via the Anthropic SDK directly
pip install "creel[semantic]"     # LinkML .yaml round-trip + RDF/SHACL validation (pyyaml, linkml, rdflib, pyshacl)

LinkML dict authoring and RDF-star/Turtle export are pure-Python and need no extra — [semantic] only adds .yaml-file authoring and downstream validators. Other extras reserve optional backends: graphdb (Neo4j/Oxigraph), er (Splink), eval (DeepEval), ocr (Tesseract). The headline ones above cover most use; pip install "creel[llm]" is an alias for the default provider (aix).

A first taste

Declare a grammar, extract a graph from prose, validate it, emit canonical JSON:

from creel import (
    GraphSpec, NodeType, EdgeType, AttrSchema, EnumDef,
    extract, validate_graph, to_canonical_json,
)

spec = GraphSpec(
    enums=(EnumDef("Currency", ("USD", "EUR")),),
    node_types=(
        NodeType("donor", attributes=(AttrSchema("name", required=True),)),
        NodeType("project", attributes=(AttrSchema("title", required=True),)),
    ),
    edge_types=(
        EdgeType("funds", subject_type="donor", object_type="project",
                 attributes=(AttrSchema("amount", range="integer", required=True, minimum=0),
                             AttrSchema("currency", range="Currency", required=True))),
    ),
)

# Deterministic pattern extractors (no LLM): regex over prose.
bindings = {
    "donor": ("regex_node", {"pattern": r"Donor:\s*(?P<name>.+)", "id_attribute": "name"}),
    "project": ("regex_node", {"pattern": r"Project:\s*(?P<title>.+)", "id_attribute": "title"}),
    "funds": ("regex_edge", {
        "pattern": r"(?P<donor>[\w ]+?) funds (?P<project>[\w ]+?) with (?P<currency>[A-Z]{3}) (?P<amount>\d+)",
        "source_id_template": "donor:{donor}", "target_id_template": "project:{project}",
        "casts": {"amount": "int"}, "exclude_groups": ("donor", "project")}),
}
src = "Donor: Gov X\nProject: Water\nGov X funds Water with USD 1000000"

g = extract(src, spec, bindings, on_missing_binding="skip")
assert validate_graph(g, spec) == []
print(to_canonical_json(g))                 # deterministic, git-diffable JSON
print(g.evidence)                            # every element traced back to its source span

With a real LLM (schema-as-extractor)

The attribute descriptions become the extraction instruction; the LLM client is injected (no provider SDK in the core). This block is self-contained:

from creel import GraphSpec, NodeType, AttrSchema, extract
from creel.extract.llm import aix_client

spec = GraphSpec(node_types=(
    NodeType("donor", description="An entity that provides funding.",
             attributes=(AttrSchema("name", required=True,
                                    description="The donor's official name."),)),
))
prose = "Donor: Foundation Alpha (ref 301). Donor: Agency Beta (ref 918)."
g = extract(prose, spec, {"donor": ("llm", {})},
            services={"llm": aix_client()}, on_missing_binding="skip")

Skills (the AI-native interface)

creel is AI-first, and that includes how you learn and drive it. It ships agent skills — SKILL.md cheat-sheets that make Claude (and any gh skill-aware agent: Copilot, Cursor, Codex, Gemini) actually good at using creel, instead of guessing from docstrings. They install with pip (bundled, offline) or gh skill:

# All consumer skills ride along with the package:
pip install creel            # creel/data/skills/* is bundled in the wheel

# …or install individually into your agent (cross-agent, pinnable):
gh skill install thorwhalen/creel creel-extract       # the end-to-end workflow
gh skill install thorwhalen/creel creel-grammar       # author a typed GraphSpec
gh skill install thorwhalen/creel creel-bindings      # choose extractor strategies
gh skill install thorwhalen/creel creel-evaluation    # pluggable verifiers (≠ ==)
gh skill install thorwhalen/creel creel-ai            # real LLM extraction
gh skill install thorwhalen/creel creel-projections   # resolve / view / export / trace
# add --agent <copilot|cursor|codex|gemini|claude> to target a host; @vX.Y.Z to pin

Once installed, just ask in plain language ("extract a graph from these documents with creel", "write a verifier for the amounts") — the matching skill triggers automatically. The skills cross-reference each other, so an agent that starts at creel-extract is handed off to creel-grammar/creel-bindings/creel-ai/creel-evaluation as needed. Prefer hand-written API calls? Scroll on — everything below still applies.

What you get

• Labeled Property Graph — attributes (funding amounts, indicator values) live on edges, which have their own identity; deterministic, git-diffable canonical JSON.
• Three extractor families behind one Extractor protocol: deterministic pattern/function, query (DuckDB SQL / JMESPath over structured sources), and LLM (schema-as-extractor, validate-retry, faithfulness gate) — plus cluster-pass (extract several coupled types in one LLM call).
• Ingestion (ingest()): route-by-format file loaders (md/csv/json/txt built-in; PDF/DOCX/XLSX/HTML via extras).
• Auditability: every node/edge/value carries a separable evidence record (provenance + grounding selector back to the exact source span + confidence). A reverse-trace index answers "which elements did this passage produce?", and a re-anchoring resolver keeps highlights valid across re-ingestion.
• Evaluation by pluggable verifiers, not ==: numeric_tolerance, set_match, graph_match (partial credit), llm_rubric (NL-defined, G-Eval), … with a corpus runner.
• Entity resolution cascade (normalize → registry → LLM), the reify edge↔node toggle, views (DOT/Mermaid/Cytoscape/tables), and export adapters (JGF, GraphML, parameterized Cypher, RDF-star).

Design & docs

The full reasoning lives in the research + design docs: start with the synthesis (decisions D1–D15), then the roadmap, the decision log, and the progress log. A worked example is in examples/.

License

MIT.