memFrame

memFrame brings a pandas-like DataFrame API to DuckDB, PostgreSQL, and ClickHouse — async-first, with an optional AI agent for natural-language data work.

Features

• Database-backed DataFrame API across DuckDB, PostgreSQL, and ClickHouse.
• Compiles every pandas-style call to backend-native SQL (DuckDB / PostgreSQL / ClickHouse) and runs it in-engine — your data never leaves the database.
• Async-first surface with sync equivalents for every operation.
• Upload from CSV, Parquet, or pandas DataFrame.
• Sync pre-existing DuckDB, PostgreSQL, or ClickHouse tables as datasets — no re-upload.
• Inspection, selection, cleaning, statistics, arithmetic, Plotly charts.
• Two-level cache: lineage audit + replayable result tables.
• Optional AI agent layer (memframe_ai) for chatting with your CSV.
• Optional Logfire observability for the AI agent layer — traces every agent run, LLM call, and tool call, plus host metrics. Opt-in via logfire_enabled (local console needs no token; bring-your-own-key for the cloud).

Installation

pip install memframe                      # core
uv add memframe                          # alt: uv
pip install "memframe[ai]"               # + Pydantic AI agent layer
uv add "memframe[ai]"                    # alt: uv
pip install "memframe[logfire]"          # + AI agent layer AND Logfire observability
pip install "memframe[ai,logfire]"       # explicit equivalent of the above

Local development from this repository:

git clone https://github.com/Debojit95/memFrame.git
cd memFrame
pip install -e ".[dev,ai]"

Quick Start

import asyncio
import pandas as pd

from memframe import MemFrame


async def main():
    mf = MemFrame(
        connection_type="local",
        connection_params={"db_path": "memFrame.duckdb"},
    )
    await mf.aconnect()

    customers = await mf.aupload_df(
        pd.DataFrame(
            {
                "id": [101, 102, 103],
                "name": ["Alice", "Bob", "Charlie"],
                "score": [95.5, 82.0, None],
                "active": [True, False, True],
            }
        ),
        filename="customers",
    )

    preview = await customers.ahead(n=5)
    print(preview["result"])

    await mf.aclose()


asyncio.run(main())

Each upload returns a dataset context; chain inspection, selection, cleaning, arithmetic, statistics, and plotting on it. Sync methods drop the a prefix (head, iloc, fillna, …).

Already have tables in your database? Register them as datasets without re-uploading:

registered = mf.register_tables()
# {'sales': [{'data_id': 'a1b2c3', 'table_name': 'orders', 'row_count': 100}, ...]}

Registered tables act like uploads — activate with set_active(data_id) and query via memFrame(). Deleting one only removes memFrame's registry entry; your real table is left untouched.

Architecture

Every pandas-style call compiles to backend-native SQL and runs in-engine — your data never leaves the database. The call path is a four-layer stack (public API → orchestrator → core SQL engine → backend adapter), with connection, ingestion, and a two-level cache as cross-cutting subsystems, plus an optional memframe_ai agent layer on top.

See the full Architecture page for the component diagram and a per-layer breakdown.

AI Agent

memframe_ai adds a Pydantic AI agent fleet on top of memFrame. After enabling it on the MemFrame instance, any dataset context can run natural-language queries that decompose into specialist tools and return typed response blocks.

import asyncio
import pandas as pd

from memframe import MemFrame


async def main():
    mf = MemFrame(
        connection_type="local",
        connection_params={"db_path": "memFrame.duckdb"},
    )
    await mf.aconnect()

    await mf.aenable_agent(
        provider="openai",
        model="gpt-5.5",
        api_key="sk-...",
    )

    ds = await mf.aupload_df(
        pd.DataFrame(
            {
                "name": ["Alice", "Bob", "Charlie"],
                "score": [95.5, 82.0, None],
            }
        ),
        filename="customers",
    )

    result = await ds.achat("fill null scores with the mean")
    print(result["answer"])
    print(result["plots"])  # any charts the agent built (id, title, spec_preview)

    await mf.aclose()


asyncio.run(main())

The agent supports OpenAI, Anthropic, Google, and Ollama. Pick a provider when you enable the agent:

await mf.aenable_agent(api_key="sk-...", provider="anthropic", model="claude-...")

Documentation

Full reference lives in docs/:

• Getting Started — connect, upload, first query.
• Connector & Connection — DuckDB / Postgres / ClickHouse wiring.
• Upload Manager — CSV / Parquet / DataFrame ingestion.
• Sync Existing Tables — register pre-existing DB tables as datasets.
• Dataset Operations — table and active-dataset management.
• Inspection · Selection · Cleaning
• Statistics · Arithmetic
• Plotting · Caching

Serve locally:

mkdocs serve