swelldb 0.1.2

Dynamic Query-Driven Table Generation with LLMs

SwellDB

Query any data — from LLMs, databases, or the web — using just DataFrames or SQL

Overview

SwellDB is a new kind of data system that enables SQL-based analytical querying over dynamically generated tables. These tables are synthesized in real-time from a combination of sources, including:

• Large Language Models (LLMs)
• Existing databases
• File formats (e.g., CSV, Parquet)
• Web search results

Unlike traditional systems operating under a closed-world assumption (queries only run on pre-loaded data), SwellDB generates tables on-demand, tailored to user-defined prompts and schemas.

This enables bridging structured SQL querying with the flexibility of unstructured data retrieval.

Figure: SwellDB Architecture

Key Features

• 🔄 Dynamic Table Generation
  Automatically synthesizes tables on-the-fly from queries and schema prompts — no need for preloaded data.

• 🌐 Multi-Source Integration
  Combines data from:

  • Large Language Models (LLMs)
  • Structured sources (e.g., CSV, SQL databases)
  • Unstructured sources (e.g., web pages, text files)
  • Web search results

• 🧠 LLM-Powered Reasoning
  Uses LLMs to:

  • Generate SQL queries over datasets
  • Extract, augment, and synthesize missing information
  • Transform unstructured text into structured tables

• 🧩 Modular & Extensible
  Easy to plug in new data sources via a clean Data Source API (structured + unstructured).

• 🧪 Fully SQL-Compatible
  Query generated tables with standard SQL — powered by Apache DataFusion.

• 🌍 Open-World Query Execution
  Go beyond what’s stored — SwellDB fetches or generates the missing pieces on demand.

• ⚡ Seamless Developer Experience
  Define tables declaratively using natural language and schema annotations. Then just write SQL.

Use Cases — Examples

• Populating relational databases from unstructured sources
  Generate tables for a relational database with DSA interview questions in SQLite.

• Ad-hoc querying across hybrid sources
  Seamlessly blend local CSVs, remote databases, LLM completions, and web results into a unified DataFrame. See example.

• Building completely new tables on-the-fly
  Dynamically generate subject-specific datasets without predefining complex ETL pipelines. See example.

🚀 Get Started

Install SwellDB

pip install swelldb

Obtain OpenAI API Key

To run the following example, you need to obtain an API key for OpenAI. You can sign up for OpenAI here. Then you can set the API keys as environment variables:

export OPENAI_API_KEY=your_openai_api_key

Create a table

from swelldb import SwellDB, OpenAILLM

swelldb: SwellDB = SwellDB(OpenAILLM(model="gpt-4o"))

table_builder = swelldb.table_builder()
table_builder.set_content("A table that contains all the US states")
table_builder.set_schema("state_name str, region str")

tbl = table_builder.build()

# Explore the table generation plan
tbl.explain()

# Create the table
table = tbl.materialize()

print(table.to_pandas())

Output

    state_name     region
0      Alabama      South
1       Alaska       West
2      Arizona       West
3     Arkansas      South
4   California       West

Querying with SQL using DataFusion

import datafusion
import pyarrow as pa

sc = datafusion.SessionContext()
sc.register_dataset("us_states", pa.dataset.dataset(table))

# Get 5 states from the South region
print(sc.sql("SELECT * FROM us_states where region = 'South' LIMIT 5"))

# Count the number of states per region
print(sc.sql("SELECT COUNT(*), region FROM us_states GROUP BY region"))

Output

DataFrame()
+------------+--------+
| state_name | region |
+------------+--------+
| Alabama    | South  |
| Arkansas   | South  |
| Delaware   | South  |
| Florida    | South  |
| Georgia    | South  |
+------------+--------+
DataFrame()
+----------+-----------+
| count(*) | region    |
+----------+-----------+
| 12       | Midwest   |
| 9        | Northeast |
| 16       | South     |
| 13       | West      |
+----------+-----------+