SilkLoom Core

A lightweight pandas accessor for batch LLM extraction.

DataFrame rows → Jinja2 prompt render → OpenAI-compatible API → repaired JSON → result DataFrame

One call — df.llm.extract(template) — concurrently sends every row to an LLM, parses the JSON response, and returns a DataFrame you can join back to the original.

Table of Contents

• Install
• Quick Start
• Configuration
  • Global Configuration
  • Per-DataFrame Setup
  • Per-Call Client
  • Priority Chain
• Extraction
  • Prompt Templates
  • Result Columns
• Cache and Audit
• Images
• Progress and Cancel
• API Reference

Install

pip install silkloom-core

# optional: progress bar
pip install silkloom-core[progress]

Importing silkloom_core registers the df.llm accessor on every DataFrame.

Quick Start

import pandas as pd
import silkloom_core

silkloom_core.configure(api_key="...", base_url="https://api.openai.com/v1")

df = pd.DataFrame({
    "title": ["A clear experiment", "A weak evaluation"],
    "abstract": ["Reliable and reproducible.", "Too small to conclude much."],
})

df = df.llm.extract(
    "Title: {{ title }}\nAbstract: {{ abstract }}\nReturn JSON with keys label and summary.",
    model="gpt-4o-mini",
    max_workers=8,
    json_mode=True,
)
# df now has: title, abstract, label, summary

extract() returns the original DataFrame with extracted columns appended. Pass join=False if you only want the extracted columns.

Configuration

SilkLoom supports three layers of client configuration, from broadest to narrowest.

Global Configuration

Configure once at the start of a script — every DataFrame can call extract() directly:

silkloom_core.configure(
    api_key="...",
    base_url="https://api.openai.com/v1",
)

# Or pass a pre-built client:
silkloom_core.configure(client=OpenAI(api_key="...", base_url="..."))

Per-DataFrame Setup

Override the global default for a specific DataFrame:

df.llm.setup(
    api_key="...",
    base_url="...",
    cache_path="special_cache.db",
)

# Chain directly into extract:
df.llm.setup(client=client).extract("...", model="gpt-4o-mini")

Per-Call Client

Override the client for a single extract() call — useful for mixing providers:

from openai import OpenAI

openai_client = OpenAI(api_key="...", base_url="https://api.openai.com/v1")
zhipu_client = OpenAI(api_key="...", base_url="https://open.bigmodel.cn/api/paas/v4")

silkloom_core.configure(client=openai_client)

# Same DataFrame, different providers:
df.llm.extract("...", model="gpt-4o")                          # → OpenAI
df.llm.extract("...", model="glm-4-flash", client=zhipu_client) # → Zhipu

Priority Chain

extract(client=...)  >  df.llm.setup(client=...)  >  silkloom_core.configure(client=...)  >  error

If no client is configured, extract() raises RuntimeError with a clear message.

Note: The SQLite cache file is created lazily — only on the first extract() call, never as a side effect of configure() or setup().

Extraction

Prompt Templates

Prompts use Jinja2 with StrictUndefined — typos in column names raise immediately instead of silently producing empty strings. Literal JSON braces ({ }) are safe; only {{ }} is treated as a template expression.

out = df.llm.extract(
    'Classify {{ text }} and return JSON like {"label": "positive", "score": 0.9}',
    model="gpt-4o-mini",
    temperature=0.1,
    max_workers=4,
    max_retries=2,
)

Result Columns

The returned DataFrame has the same index as the input. Column semantics:

Malformed JSON is repaired with json_repair before parsing.

Cache and Audit

Every API call is recorded in a SQLite database. Successful responses (ok = 1) are reused as cache hits on subsequent runs with the same request. Failed requests are also stored for debugging but are retried next time.

df.llm.setup(cache_path="cache/llm.sqlite").extract(...)

The cache table schema:

To start fresh: delete the SQLite file or use a different cache_path.

Images

Pass image_column for local file paths, HTTP(S) URLs, or data:image/... URLs. Local files are auto-encoded as base64 data URLs with MIME detection. A single cell can hold a list of images for multi-image input.

out = df.llm.extract(
    "Extract fields from this receipt and return JSON.",
    image_column="receipt_path",
    model="gpt-4o-mini",
)

Rows with missing image values (NaN, None) fall back to text-only prompts.

Progress and Cancel

Progress bar — tqdm is used when verbose=True (default). If tqdm isn't installed, it degrades silently.

Callback — for UI integration:

def progress(done, total):
    print(f"{done}/{total}")

out = df.llm.extract("Analyze {{ text }}", progress_callback=progress)

Cancel — from another thread:

df.llm.cancel()

Queued work is cancelled where possible. Running rows stop before their next retry. Already-completed results are preserved in the returned DataFrame.

API Reference

silkloom_core.configure(...)

df.llm.setup(...)

Same parameters as configure(). Returns self for chaining.

df.llm.extract(prompt_template, *, ...)

df.llm.cancel()

No parameters. Signals cancellation to all in-flight work.