polar-llama 0.6.3

Polar Llama

Overview

Polar Llama is a Python library designed to enhance the efficiency of making parallel inference calls to the ChatGPT API using the Polars dataframe tool. This library enables users to manage multiple API requests simultaneously, significantly speeding up the process compared to serial request handling.

Key Features

• Parallel Inference: Send multiple inference requests in parallel to the ChatGPT API without waiting for each individual request to complete.
• Integration with Polars: Utilizes the Polars dataframe for organizing and handling requests, leveraging its efficient data processing capabilities.
• Easy to Use: Simplifies the process of sending queries and retrieving responses from the ChatGPT API through a clean and straightforward interface.
• Multi-Message Support: Create and process conversations with multiple messages in context, supporting complex multi-turn interactions.
• Multiple Provider Support: Works with OpenAI, Anthropic, Gemini, Groq, and AWS Bedrock models, giving you flexibility in your AI infrastructure.
• Structured Outputs: Define response schemas using Pydantic models for type-safe, validated LLM outputs returned as Polars Structs with direct field access.
• Vector Similarity: Rust-powered similarity metrics (cosine, dot product, Euclidean distance) for high-performance vector operations.
• Approximate Nearest Neighbor Search: HNSW algorithm for fast semantic search and recommendations at scale.
• Prompt Optimization: A DSPy-style optimization engine (Signature, Predict, BootstrapFewShot, InstructionOptimizer) that tunes instructions and few-shot demos against your labeled data using parallel batched evaluation.
• Tool Use / MCP: Dataframe-native tool calling — LLMs emit tool calls as structured output, and execute_tool_calls runs every call of every row in parallel against an MCP server or a Python callable. See docs/TOOL_USE.md.
• Prompt Caching: Provider-native prompt caching (Anthropic 5m/1h cache_control) to share a cached system prefix across rows — pass cache=True with a system_prompt.

Installation

To install Polar Llama, you can use pip:

pip install polar-llama

Alternatively, for development purposes, you can install from source:

maturin develop

Example Usage

Here's how you can use Polar Llama to send multiple inference requests in parallel:

import polars as pl
from polar_llama import Provider
import dotenv

dotenv.load_dotenv()

# Example questions
questions = [
    'What is the capital of France?',
    'What is the difference between polars and pandas?'
]

# Creating a dataframe with questions
df = pl.DataFrame({'Questions': questions})

# Using the fluent .llama namespace (recommended)
df = df.with_columns(
    answer=pl.col('Questions').llama.inference_async(
        provider=Provider.OPENAI,
        model='gpt-4o-mini'
    )
)

# Alternative: Using functional API
from polar_llama import string_to_message, inference_async

df = df.with_columns(
    prompt=string_to_message(pl.col("Questions"), message_type='user')
)
df = df.with_columns(
    answer=inference_async(pl.col('prompt'), provider=Provider.OPENAI, model='gpt-4o-mini')
)

Multi-Message Conversations

Polar Llama supports multi-message conversations, allowing you to maintain context across multiple turns:

import polars as pl
from polar_llama import combine_messages, inference_messages
import dotenv

dotenv.load_dotenv()

# Create a dataframe with system prompts and user questions
df = pl.DataFrame({
    "system_prompt": [
        "You are a helpful assistant.",
        "You are a math expert."
    ],
    "user_question": [
        "What's the weather like today?",
        "Solve x^2 + 5x + 6 = 0"
    ]
})

# Using .llama namespace (recommended)
df = df.with_columns([
    pl.col("system_prompt").llama.to_message(role="system").alias("system_message"),
    pl.col("user_question").llama.to_message(role="user").alias("user_message")
])

# Combine into conversations
df = df.with_columns(
    conversation=combine_messages(pl.col("system_message"), pl.col("user_message"))
)

# Send to model and get responses
df = df.with_columns(
    response=inference_messages(pl.col("conversation"), provider="openai", model="gpt-4")
)

AWS Bedrock Support

Polar Llama now supports AWS Bedrock models. To use Bedrock, ensure you have AWS credentials configured (via AWS CLI, environment variables, or IAM roles):

import polars as pl
from polar_llama import string_to_message, inference_async
import dotenv

dotenv.load_dotenv()

# Example questions
questions = [
    'What is the capital of France?',
    'Explain quantum computing in simple terms.'
]

# Creating a dataframe with questions
df = pl.DataFrame({'Questions': questions})

# Adding prompts to the dataframe
df = df.with_columns(
    prompt=string_to_message(pl.col("Questions"), message_type='user')
)

# Using AWS Bedrock with Claude model
df = df.with_columns(
    answer=inference_async(pl.col('prompt'), provider='bedrock', model='anthropic.claude-3-haiku-20240307-v1:0')
)

Structured Outputs with Pydantic

Polar Llama supports structured outputs using Pydantic models. Define your response schema as a Pydantic BaseModel, and the LLM will return validated, type-safe data as a Polars Struct:

import polars as pl
from polar_llama import inference_async, Provider
from pydantic import BaseModel

# Define your response schema
class MovieRecommendation(BaseModel):
    title: str
    genre: str
    year: int
    reason: str

# Create a dataframe
df = pl.DataFrame({
    'prompt': ['Recommend a great sci-fi movie from the 2010s']
})

# Get structured output
df = df.with_columns(
    recommendation=inference_async(
        pl.col('prompt'),
        provider=Provider.OPENAI,
        model='gpt-4o-mini',
        response_model=MovieRecommendation
    )
)

# Access struct fields directly!
print(df['recommendation'].struct.field('title')[0])  # "Interstellar"
print(df['recommendation'].struct.field('year')[0])   # 2014

Key Features:

• Type Safety: Responses are validated against your Pydantic schema
• Direct Field Access: Use .struct.field('field_name') to access individual fields
• Error Handling: Built-in _error, _details, and _raw fields for graceful error handling
• Works Everywhere: Compatible with inference_async(), inference(), and inference_messages()
• Multi-Provider: Works with OpenAI, Anthropic, Groq, Gemini, and Bedrock

Error Handling:

# Check for errors in responses
error = df['recommendation'].struct.field('_error')[0]
if error:
    print(f"Error: {error}")
    print(f"Details: {df['recommendation'].struct.field('_details')[0]}")
    print(f"Raw response: {df['recommendation'].struct.field('_raw')[0]}")

Streaming Responses

inference_stream() streams completions token-by-token via an on_token(row_index, delta) callback, while still returning a plain DataFrame column — there's no separate iterator to manage. Each row's result is a Struct{text: Utf8, finished: Boolean}: finished=True means the stream completed normally; finished=False covers any interruption (a provider error mid-stream, a dropped connection, Ctrl-C, or an on_token callback that raises) and comes with a RuntimeWarning plus whatever partial text had arrived — the DataFrame itself is never left in a broken state.

import polars as pl
from polar_llama import inference_stream, Provider

df = pl.DataFrame({
    'prompt': ["Tell me a short joke", "Say hello in French"]
})

df = df.with_columns(
    response=inference_stream(
        pl.col('prompt'),
        provider=Provider.OPENAI,
        model='gpt-4o-mini',
        on_token=lambda i, d: print(d, end=""),
    )
)

print(df.select(
    pl.col('response').struct.field('text'),
    pl.col('response').struct.field('finished'),
))

Streaming is text-only: pass response_model and you'll get a clear ValueError telling you to use inference_async() for structured output instead. OpenAI, Groq, and Anthropic stream natively over SSE; Gemini and Bedrock fall back to a single "full response, then done" delta so every provider works with the same API.

Resumable Runs (Checkpointing)

Large jobs fail mid-run — rate limits, network blips, provider outages. Pass checkpoint="path" and completed rows are persisted to a sidecar store as the run progresses; re-run the same code and it resumes, skipping rows that already finished so you pay for roughly one full pass, not two.

from polar_llama import inference_async, Provider

out = df.with_columns(
    answer=inference_async(
        pl.col("prompt"),
        provider=Provider.OPENAI,
        model="gpt-4o-mini",
        checkpoint="run1.ckpt",   # directory of append-only Parquet parts
    )
)
# If this dies at 50%, just run it again — the first 50% is served from
# the checkpoint and only the remaining rows hit the API.

The checkpoint key hashes the row input and the run config (provider, model, prompt, response schema, endpoint), so changing any of them invalidates old entries automatically — you never get a stale result from a different configuration. Failed rows are stored as failed and retried on resume by default (Checkpoint(path, retry_failed=False) to return the stored error instead). Works with structured outputs and tool-use passes. See docs/design/CHECKPOINTING.md.

Per-Row Usage & Cost

Pass usage=True to get token counts, latency, and estimated cost per row as a struct column — useful for invoicing against LLM spend or surfacing usage to users.

out = df.with_columns(
    r=inference_async(pl.col("prompt"), provider=Provider.OPENAI,
                      model="gpt-4o-mini", usage=True)
)
# r is Struct{response, usage: Struct{input_tokens, output_tokens,
#                                     cached_tokens, latency_ms, cost_usd}}
totals = out.select(
    cost=pl.col("r").struct.field("usage").struct.field("cost_usd").sum(),
    in_tok=pl.col("r").struct.field("usage").struct.field("input_tokens").sum(),
)

cost_usd is computed from a packaged, overridable price table (polar_llama/pricing.py); pass price_table= (a dict or path) or call register_model_price(...) to add/override models. Unknown models yield cost_usd = null with a one-time warning. Usage is captured for OpenAI, Groq, Anthropic, Gemini, and Bedrock; the MLX local engine reports tokens + latency with cost_usd = 0.0. usage=False (the default) is unchanged.

Duplicate Collapsing & a Persistent Response Cache

Two related, separately-gated features. dedupe=True collapses exact-duplicate rows within one run — each unique input is sent once and the result is fanned back out to every row that asked for it, zero extra I/O:

out = df.with_columns(answer=inference_async(pl.col("prompt"), model="gpt-4o-mini", dedupe=True))

response_cache="path" adds a persistent, cross-job store (implies dedupe=True automatically): identical requests from a different run, even a different process, reuse a prior result instead of re-calling the backend.

from polar_llama import ResponseCache, DedupeStats

stats = DedupeStats()
out = df.with_columns(
    answer=inference_async(
        pl.col("prompt"),
        model="gpt-4o-mini",
        response_cache=ResponseCache("responses.cache", ttl="24h"),
        dedupe_stats=stats,
    )
)
print(stats.hit_rate, stats.rows_collapsed, stats.calls_made)

Only successful results are ever persisted — a failed row always recomputes on the next run. ttl= expires entries lazily on read; call ResponseCache(...).prune() to compact the store and reclaim expired entries, or .clear() to invalidate everything. Not currently supported together with checkpoint= or usage=True (raises ValueError — see docs/design/RESPONSE_CACHE.md for the full interop rules and why).

Local Inference (Apple Silicon / MLX)

Run inference on-device on Apple Silicon instead of a provider API — no keys, no network — via mlx-lm. Install the extra (Apple Silicon, Python ≥ 3.10):

pip install "polar-llama[local]"

import polars as pl
import polar_llama  # registers the .llama namespace

df = pl.DataFrame({"ticket": [
    "My laptop won't turn on even when it's plugged in.",
    "I forgot my password and I'm locked out of my account.",
]})

tagged = df.with_columns(
    tag=pl.col("ticket").llama.inference_local(
        model="mlx-community/gemma-3n-E4B-it-lm-4bit",
        system="Classify the ticket as exactly one of: Hardware, Account, Billing.",
        engine="in_process",   # on-device mlx-lm, batched over the column
        max_tokens=16,
        temperature=0.0,
    )
)

Two engines are available:

• engine="in_process" — batched generation directly via mlx-lm (BatchGenerator / batch_generate); nothing to run separately.
• engine="server" (default) — routes the existing async fan-out to a local OpenAI-compatible endpoint (mlx_lm.server) via OPENAI_BASE_URL.

Collapsed prefill. When rows share a long common prefix (a shared system prompt, or few-shot demos), set POLAR_LLAMA_LOCAL_COLLAPSE=1 to compute that prefix once instead of re-prefilling it per row — a large speedup for prefill-bound workloads, at parity-verified output. Hybrid Gemma 3n / Gemma 4 models need the mlx-lm #1384 batched-RoPE fix, which Polar Llama applies automatically. See docs/local_mlx_backend.md.

Prompt Optimization (DSPy-style)

Polar Llama ships a lightweight prompt optimization engine inspired by DSPy. Declare your task as a Signature, wrap it in a Predict module, and let an optimizer tune the prompt against your labeled DataFrame — every candidate is evaluated with one parallel, batched inference call:

import polars as pl
from polar_llama import Predict, Signature, BootstrapFewShot, InstructionOptimizer, evaluate

# 1. Declare the task
module = Predict(
    Signature("question -> answer", instructions="Answer concisely."),
    provider="openai",
    model="gpt-4o-mini",
)

# 2. Labeled training data
trainset = pl.DataFrame({
    "question": ["What is 2+2?", "Capital of France?", "Largest planet?"],
    "answer": ["4", "Paris", "Jupiter"],
})

# 3. A metric: (gold row, prediction) -> bool | float
def exact_match(example, prediction):
    return example["answer"].strip().lower() == (prediction["answer"] or "").strip().lower()

# 4a. Bootstrap few-shot demos from rows the model already gets right
compiled = BootstrapFewShot(metric=exact_match, max_demos=4).compile(module, trainset)

# 4b. Or search for better instructions (COPRO-style)
optimizer = InstructionOptimizer(metric=exact_match, n_candidates=4)
compiled = optimizer.compile(module, trainset)
print(optimizer.history)  # [(instructions, score), ...]

# 5. Run the optimized module on new data — outputs land in pred_* columns
result = compiled(pl.DataFrame({"question": ["What is 3+3?"]}))
print(result["pred_answer"])

# Score any module against a labeled set
print(evaluate(compiled, trainset, exact_match).score)

Output fields can be typed and described for stronger structured outputs:

from polar_llama import OutputField

sig = Signature(
    "review -> sentiment, confidence",
    instructions="Classify the sentiment of the review.",
    outputs={
        "sentiment": OutputField(desc="one of: positive, negative, neutral"),
        "confidence": OutputField(desc="confidence from 0.0 to 1.0", dtype=float),
    },
)

Optimize entirely on-device. Drive the optimizer with a local model instead of a provider by passing an inference_fn — no API keys, no network:

from polar_llama.local import make_local_inference_fn

# Backs Predict/BootstrapFewShot/InstructionOptimizer with on-device gemma-3n
# (mlx-lm). Collapsed prefill (on by default) shares the system+demos prefix
# across rows — the dominant speedup during tuning, where few-shot demos make
# that prefix ~80% of every prompt (~2.8x on a full tuning schedule, at
# identical output). Requires the [local] extra.
fn = make_local_inference_fn("mlx-community/gemma-3n-E4B-it-lm-4bit")
module = Predict(Signature("note -> category", instructions="Classify the note."), inference_fn=fn)
compiled = InstructionOptimizer(metric=exact_match).compile(module, trainset)

Vector Embeddings

Polar Llama provides parallelized, memory-efficient embedding generation for converting text into vector representations. This is useful for semantic search, clustering, and similarity analysis:

import polars as pl
from polar_llama import embedding_async, Provider

# Create a dataframe with text
df = pl.DataFrame({
    'text': [
        'Machine learning is transforming AI',
        'Natural language processing enables text understanding',
        'Deep learning uses neural networks'
    ]
})

# Generate embeddings in parallel
df = df.with_columns(
    embeddings=embedding_async(
        pl.col('text'),
        provider=Provider.OPENAI,
        model='text-embedding-3-small'  # 1536 dimensions
    )
)

# Access embedding dimensions
df = df.with_columns(
    dimensions=pl.col('embeddings').list.len()
)

print(df['dimensions'][0])  # 1536

Using the .llama namespace:

df = df.with_columns(
    embeddings=pl.col('text').llama.embedding(
        provider=Provider.OPENAI,
        model='text-embedding-3-small'
    )
)

Supported Models:

• OpenAI: text-embedding-3-small (1536 dims), text-embedding-3-large (3072 dims)
• Gemini: text-embedding-004 (768 dims)
• AWS Bedrock: amazon.titan-embed-text-v1 (1536 dims)

Key Features:

• Parallel Processing: All embeddings are generated concurrently using async/await for maximum performance
• Memory Efficient: Streaming approach minimizes memory footprint
• Type Safety: Returns List[Float64] for seamless integration with Polars operations
• Null Handling: Gracefully handles null values in input

Practical Example - Semantic Similarity:

from polar_llama import cosine_similarity

# Generate embeddings
df = df.with_columns(
    embeddings=embedding_async(pl.col('text'), provider=Provider.OPENAI)
)

# Calculate similarity between first document and all others
query_emb = df["embeddings"][0]
df = df.with_columns(
    similarity=cosine_similarity(
        pl.lit([query_emb]),
        pl.col('embeddings')
    )
)

print(df.select(['text', 'similarity']))

Vector Similarity and Approximate Nearest Neighbor Search

Polar Llama includes high-performance Rust-powered vector similarity operations and approximate nearest neighbor (ANN) search capabilities for semantic search, recommendations, and clustering:

Similarity Metrics:

from polar_llama import cosine_similarity, dot_product, euclidean_distance

df = pl.DataFrame({
    "vec1": [[1.0, 0.0, 0.0], [1.0, 2.0, 3.0]],
    "vec2": [[1.0, 0.0, 0.0], [2.0, 4.0, 6.0]]
})

# Calculate different similarity metrics
df = df.with_columns([
    cosine_similarity(pl.col("vec1"), pl.col("vec2")).alias("cosine_sim"),
    dot_product(pl.col("vec1"), pl.col("vec2")).alias("dot_prod"),
    euclidean_distance(pl.col("vec1"), pl.col("vec2")).alias("distance")
])

# Using .llama namespace (alternative)
df = df.with_columns(
    cosine_sim=pl.col("vec1").llama.cosine_similarity(pl.col("vec2"))
)

HNSW Approximate Nearest Neighbor Search:

Fast k-nearest neighbor search using the HNSW (Hierarchical Navigable Small World) algorithm:

from polar_llama import knn_hnsw, embedding_async, Provider

# Create corpus of documents
corpus = pl.DataFrame({
    "doc": ["AI research", "cooking tips", "machine learning", "recipes"]
}).with_columns(
    embedding=embedding_async(pl.col("doc"), provider=Provider.OPENAI)
)

# Create query
query = pl.DataFrame({
    "query": ["artificial intelligence"]
}).with_columns(
    query_emb=embedding_async(pl.col("query"), provider=Provider.OPENAI),
    corpus_emb=pl.lit([corpus["embedding"].to_list()])
).with_columns(
    neighbors=knn_hnsw(
        pl.col("query_emb"),
        pl.col("corpus_emb").list.first(),
        k=2  # Find 2 nearest neighbors
    )
)

# Get nearest neighbor documents
indices = query["neighbors"][0]
print(corpus[indices]["doc"])  # ['AI research', 'machine learning']

Metadata-Enhanced Search:

Combine taxonomy filtering with vector search for precise, context-aware results:

from polar_llama import tag_taxonomy, embedding_async, knn_hnsw, Provider

# Define taxonomy for metadata
taxonomy = {
    "category": {
        "description": "Content category",
        "values": {
            "technology": "Tech and programming content",
            "cooking": "Food and recipe content"
        }
    }
}

# Use with_columns() for parallel execution
corpus = corpus.with_columns([
    tag_taxonomy(pl.col("content"), taxonomy, provider=Provider.ANTHROPIC)
        .alias("tags"),
    embedding_async(pl.col("content"), provider=Provider.OPENAI)
        .alias("embedding")
])

# Note: with_columns() runs operations in parallel
# Speedup depends on operation durations (best when similar duration)
# Each operation also internally parallelizes all API calls across all documents

# Extract category
corpus = corpus.with_columns(
    category=pl.col("tags").struct.field("category").struct.field("value")
)

# STEP 1: Filter by metadata (category = technology)
tech_docs = corpus.filter(pl.col("category") == "technology")

# STEP 2: Semantic search within filtered subset
query = query.with_columns(
    tech_corpus=pl.lit([tech_docs["embedding"].to_list()])
).with_columns(
    tech_neighbors=knn_hnsw(pl.col("query_emb"), pl.col("tech_corpus").list.first(), k=3)
)

# Results are guaranteed to be technology-related AND semantically relevant

Key Features:

• Blazing Fast: Rust-powered similarity calculations with zero-copy operations
• Multiple Metrics: Cosine similarity, dot product, and Euclidean distance
• HNSW Algorithm: State-of-the-art approximate nearest neighbor search
• Parallel Processing: Vectorized operations for maximum performance
• Metadata Filtering: Combine structured and semantic search

See docs/VECTOR_SIMILARITY_AND_ANN.md for complete documentation and advanced examples.

Tool Use and MCP

Polar Llama supports tool calling without hiding an agent loop inside your rows. The loop is unrolled into the dataframe: the LLM emits tool calls as structured output, execute_tool_calls runs every call of every row concurrently (against an MCP server or any Python callable), and a second inference pass synthesizes the results — every intermediate step is an ordinary column.

from polar_llama import (
    mcp_tools, tools_to_response_model, execute_tool_calls,
    tool_results_to_message, combine_messages, inference_messages, Provider,
)

tools = mcp_tools("http://localhost:8811/mcp")     # tools/list introspection
ToolCalls = tools_to_response_model(tools)          # emission schema

df = (
    df
    # 1. Emit: the LLM parameterizes N calls per row (structured output)
    .with_columns(calls=pl.col("meal").llama.inference_async(
        provider=Provider.OPENAI, model="gpt-4o-mini", response_model=ToolCalls))
    # 2. Execute: all calls across all rows, in parallel, errors as data
    .with_columns(results=execute_tool_calls(
        pl.col("calls"), transport="http://localhost:8811/mcp", tools=tools))
    # 3. Synthesize: fold results back through a second inference pass
    .with_columns(answer=inference_messages(
        combine_messages(
            pl.col("meal").llama.to_message(role="user"),
            tool_results_to_message(pl.col("results")),
        ),
        provider=Provider.OPENAI, model="gpt-4o-mini"))
)

See docs/TOOL_USE.md for the full guide and docs/design/MCP_TOOL_INTEGRATION.md for the design rationale.

Benefits

• Speed: Processes multiple queries in parallel, drastically reducing the time required for bulk query handling.
• Scalability: Scales efficiently with the increase in number of queries, ideal for high-demand applications.
• Ease of Integration: Integrates seamlessly into existing Python projects that utilize Polars, making it easy to add parallel processing capabilities.
• Context Preservation: Maintain conversation context with multi-message support for more natural interactions.
• Provider Flexibility: Choose from multiple LLM providers based on your needs and access.
• Type Safety: Get validated, structured outputs using Pydantic schemas for reliable data extraction.

Testing

Polar Llama includes a comprehensive test suite that validates parallel execution, provider support, and core functionality.

Setup:

1. Copy .env.example to .env and add your API keys:

   cp .env.example .env
   # Edit .env and add your provider API keys
   

2. Install test dependencies:

   pip install -r tests/requirements.txt
   

Run Python tests:

pytest tests/ -v

Run Rust tests:

cargo test --test model_client_tests -- --nocapture

Tests automatically detect configured providers and only run tests for those with valid API keys. See tests/README.md for detailed testing documentation.

Contributing

We welcome contributions to Polar Llama! If you're interested in improving the library or adding new features, please feel free to fork the repository and submit a pull request.

License

Polar Llama is released under the MIT license. For more details, see the LICENSE file in the repository.

Roadmap

• Multi-Message Support: Support for multi-message conversations to maintain context.
• Multiple Provider Support: Support for different LLM providers (OpenAI, Anthropic, Gemini, Groq, AWS Bedrock).
• Structured Data Outputs: Add support for structured data outputs using Pydantic models with type validation and Polars Struct returns.
• Tool Use / MCP: Batch-parallel tool-call emission and execution with MCP support (see docs/TOOL_USE.md).
• Streaming Responses: Support for streaming responses from LLM providers (inference_stream(), see above).