DataFrameIt

English · Português · Español

Enrich DataFrames with LLMs, simply and in a structured way.

DataFrameIt processes text in DataFrames using Large Language Models (LLMs) and extracts structured information defined by Pydantic models.

Full Documentation | LLM Reference

Installation

pip install dataframeit[openai]       # OpenAI (default provider)
pip install dataframeit[google]       # Google Gemini
pip install dataframeit[anthropic]    # Anthropic Claude
pip install dataframeit[groq]         # Groq
pip install dataframeit[codex]        # Official Codex SDK (experimental)
pip install dataframeit[claude-code]  # Claude Code via the Claude Agent SDK
pip install dataframeit[all]          # the providers above except Codex, plus web search, Polars and Excel

Optional extras: search (Tavily), search-exa (Exa), search-all, polars, excel.

Set up provider authentication:

export OPENAI_API_KEY="your-key"  # or GOOGLE_API_KEY, ANTHROPIC_API_KEY, GROQ_API_KEY

The experimental codex provider is optional, is not part of the all extra, uses the bundled runtime and requires local file-based authentication. See the installation docs to set up the extra and the credentials.

Quick Example

from pydantic import BaseModel
from typing import Literal
import pandas as pd
from dataframeit import dataframeit

# 1. Define what to extract
class Sentiment(BaseModel):
    sentiment: Literal['positive', 'negative', 'neutral']
    confidence: Literal['high', 'medium', 'low']

# 2. Your data
df = pd.DataFrame({
    'text': [
        'Excellent product! Exceeded my expectations.',
        'Terrible service, never buying again.',
        'Delivery was fine, product is average.'
    ]
})

# 3. Process!
result = dataframeit(df, Sentiment, "Analyze the sentiment of the text.")
print(result)

Output:

With track_tokens=True, the default, the output also has the token usage columns _input_tokens, _cached_input_tokens, _output_tokens and _reasoning_tokens, left out of the table above.

Field and class names are arbitrary; the examples in the example/ notebooks use Portuguese ones.

Features

• Multiple providers: OpenAI, Google Gemini, Anthropic and Groq with their own extras, any other LangChain provider (Cohere, Mistral, Vertex AI, Bedrock, Azure) by installing its package, plus Claude Code and Codex through their official SDKs
• Multiple input types: pandas or Polars DataFrame and Series, list, dict
• Structured output: Pydantic validation; with the LangChain providers, a rejected answer goes back to the model with the error
• Resilience: Automatic retry with exponential backoff, and periodic checkpoints (batch_size + checkpoint_path) to resume long runs
• Performance: Continuous parallel queue that halves the rows in flight on rate limits, configurable rate limiting, and a per-row callback (on_row_done) that runs on the worker thread
• Web search: Tavily or Exa, per field or per group of fields, with conditional fields (condition, depends_on)
• Tracking: Token usage, search credits and throughput metrics

Per-Field Configuration

Set field-specific prompts and search parameters using json_schema_extra:

from pydantic import BaseModel, Field

class DrugInfo(BaseModel):
    # Field with the default prompt
    active_ingredient: str = Field(description="Active ingredient of the drug")

    # Field with a custom prompt (replaces the base prompt)
    rare_disease: str = Field(
        description="Rare disease classification",
        json_schema_extra={
            "prompt": "Search Orphanet (orpha.net). Analyze: {texto}"
        }
    )

    # Field with an additional prompt (appended to the base prompt)
    conitec_assessment: str = Field(
        description="CONITEC assessment",
        json_schema_extra={
            "prompt_append": "Search ONLY the CONITEC website (gov.br/conitec)."
        }
    )

    # Field with custom search parameters
    clinical_trials: str = Field(
        description="Relevant clinical trials",
        json_schema_extra={
            "prompt_append": "Search for recent clinical trials.",
            "search_depth": "advanced",
            "max_results": 10
        }
    )

# Requires search_per_field=True
result = dataframeit(
    df,
    DrugInfo,
    "Analyze the drug: {texto}",
    use_search=True,
    search_per_field=True,
)

The full list of per-field options (including max_search_calls, condition and depends_on) is in the web search guide.

Documentation

• Quickstart
• Guides
• API Reference
• LLM Reference - Compact page optimized for coding assistants
• FAQ
• Changelog

Examples

See the example/ folder for Jupyter notebooks with complete use cases.

Contributing

See CONTRIBUTING.md.

License

MIT