memFrame

memFrame brings a pandas-like DataFrame API to DuckDB, PostgreSQL, and ClickHouse — async-first, with an optional AI agent for natural-language data work.

Features

• Database-backed DataFrame API across DuckDB, PostgreSQL, and ClickHouse.
• Compiles every pandas-style call to backend-native SQL (DuckDB / PostgreSQL / ClickHouse) and runs it in-engine — your data never leaves the database.
• Async-first surface with sync equivalents for every operation.
• Upload from CSV, Parquet, or pandas DataFrame.
• Pandas-style mf.DataFrame(...) constructor for dicts, frames, or existing dataset contexts.
• Database-native indexes: DuckDB ART, PostgreSQL B-tree/partial/covering, ClickHouse projection indexes (no full data copy) with explicit skipping-index types, plus a guarded ClickHouse ORDER BY rebuild.
• Sync pre-existing DuckDB, PostgreSQL, or ClickHouse tables as datasets — no re-upload.
• Two-level cache: lineage audit + replayable result tables.
• Optional AI agent layer (memframe_ai) for chatting with your CSV.
• Optional Logfire observability for the AI agent layer — traces every agent run, LLM call, and tool call, plus host metrics. Opt-in via logfire_enabled (local console needs no token; bring-your-own-key for the cloud).

Installation

pip install memframe                      # core
uv add memframe                          # alt: uv
pip install "memframe[ai]"               # + Pydantic AI agent layer
uv add "memframe[ai]"                    # alt: uv
pip install "memframe[logfire]"          # + AI agent layer AND Logfire observability
pip install "memframe[ai,logfire]"       # explicit equivalent of the above

Local development from this repository:

git clone https://github.com/Debojit95/memFrame.git
cd memFrame
uv sync --extra dev --extra ai
uv build

Quick Start

import asyncio

import memframe as mf


async def main():
    await mf.aconnect(
        connection_type="local",
        connection_params={"db_path": "memFrame.duckdb"},
    )

    customers = await mf.aDataFrame(
        {
            "id": [101, 102, 103],
            "name": ["Alice", "Bob", "Charlie"],
            "score": [95.5, 82.0, None],
            "active": [True, False, True],
        },
        filename="customers",
    )

    preview = await customers.ahead(n=5)
    print(preview)

    await mf.aclose()


asyncio.run(main())

memframe itself is the connection facade: mf.connect(...) / await mf.aconnect(...) configure one default connection, and every instance method (upload_df, list_tables, set_active, mean, chat, …) is available directly on the module. Each upload returns a dataset context; chain inspection, selection, cleaning, arithmetic, statistics, and plotting on it. Sync methods drop the a prefix (head, iloc, fillna, …).

Need more than one connection? Use the class directly:

from memframe import MemFrame

analytics = MemFrame(
    connection_type="local",
    connection_params={"db_path": "analytics.duckdb"},
)
analytics.connect()

Build a dataset from in-memory data with the pandas-style constructor:

dataset = mf.DataFrame({"name": ["Alice", "Bob"], "score": [95.5, 82.0]})
dataset = mf.DataFrame(existing_dataset)   # snapshot into a new dataset

Already have tables in your database? Register them as datasets without re-uploading (after mf.connect(...)):

registered = mf.register_tables()
# {'sales': [{'data_id': 'a1b2c3', 'table_name': 'orders', 'row_count': 100}, ...]}

Registered tables act like uploads — activate with set_active(data_id) and query via memFrame(). Deleting one only removes memFrame's registry entry; your real table is left untouched.

Architecture

Every pandas-style call compiles to backend-native SQL and runs in-engine — your data never leaves the database. The call path is a four-layer stack (public API → orchestrator → core SQL engine → backend adapter), with connection, ingestion, and a two-level cache as cross-cutting subsystems, plus an optional memframe_ai agent layer on top.

See the full Architecture page for the component diagram and a per-layer breakdown.

AI Agent

memframe_ai adds a Pydantic AI agent fleet on top of memFrame. After enabling it on the MemFrame instance, any dataset context can run natural-language queries that decompose into specialist tools and return typed response blocks.

import asyncio

import memframe as mf


async def main():
    await mf.aconnect(
        connection_type="local",
        connection_params={"db_path": "memFrame.duckdb"},
    )

    await mf.aenable_agent(
        provider="openai",
        model="gpt-5.5",
        api_key="sk-...",
    )

    ds = await mf.aDataFrame(
        {
            "name": ["Alice", "Bob", "Charlie"],
            "score": [95.5, 82.0, None],
        },
        filename="customers",
    )

    result = await ds.achat("fill null scores with the mean")
    print(result["answer"])
    print(result["plots"])  # any charts the agent built (id, title, spec_preview)

    await mf.aclose()


asyncio.run(main())

The agent supports OpenAI, Anthropic, Google, and Ollama. Pick a provider when you enable the agent:

await mf.aenable_agent(api_key="sk-...", provider="anthropic", model="claude-...")

Documentation

Full reference lives in docs/:

• Getting Started — connect, upload, first query.
• Connector & Connection — DuckDB / Postgres / ClickHouse wiring.
• Upload Manager — CSV / Parquet / DataFrame ingestion.
• Sync Existing Tables — register pre-existing DB tables as datasets.
• Dataset Operations — table and active-dataset management.
• Inspection · Selection · Filter · Clean
• Statistics · Arithmetic · Comparison · Cumulative
• Sorting · Index · Reshape · Merge · Transform
• Plotting · Caching

Serve locally:

mkdocs serve