Skip to main content

lotus

Project description

LOTUS: A Query Engine For Processing Data with LLMs

Colab Demo Arxiv Documentation Status PyPI - Python Version PyPI GitHub license

Easily build knowledge-intensive LLM applications that reason over your data with LOTUS!

LOTUS (LLMs Over Tables of Unstructured and Structured Data) provides a declarative programming model and an optimized query engine for serving powerful reasoning-based query pipelines over structured and unstructured data! We provide a simple and intuitive Pandas-like API, that implements semantic operators.

Key Concept: The Semantic Operator Model

LOTUS' implements is the semantic operator programming model. Semantic operators as declarative transformations on one or more datasets, parameterized by a natural language expression, that can be implemented by a variety of AI-based algorithms. Semantic operators seamlessly extend the relational model, operating over tables that may contain traditional structured data as well as unstructured fields, such as free-form text. These composable, modular language- based operators allow you to write AI-based pipelines with high-level logic, leaving the rest of the work to the query engine! Each operator can be implemented and optimized in multiple ways, opening a rich space for execution plans, similar to relational operators. To learn more about the semantic operator model, read the full research paper.

LOTUS offers a number of semantic operators in a Pandas-like API, some of which are described below. To learn more about semantic operators provided in LOTUS, check out the full documentation, run the colab tutorial, or you can also refer to these examples.

Operator Description
sem_map Map each record using a natural language projection
sem_filter Keep records that match the natural language predicate
sem_agg Performs a natural language aggregation across all records (e.g. for summarization)
sem_topk Order the records by some natural langauge sorting criteria
sem_join Join two datasets based on a natural language predicate
sem_dedup Deduplicate records based on semantic similarity
sem_index Create a semantic similarity index over a text column
sem_search Perform top-k search the over a text column

Installation

conda create -n lotus python=3.10 -y
conda activate lotus
pip install lotus-ai

Quickstart

If you're already familiar with Pandas, getting started will be a breeze! Below we provide a simple example program using the semantic join operator. The join, like many semantic operators, are specified by langex (natural language expressions), which the programmer uses to specify the operation. Each langex is parameterized by one or more table columns, denoted in brackets. The join's langex serves as a predicate and is parameterized by a right and left join key.

import pandas as pd
import lotus
from lotus.models import LM

# configure the LM, and remember to export your API key
lm = LM(model="gpt-4o-mini")
lotus.settings.configure(lm=lm)

# create dataframes with course names and skills
courses_data = {
    "Course Name": [
        "History of the Atlantic World",
        "Riemannian Geometry",
        "Operating Systems",
        "Food Science",
        "Compilers",
        "Intro to computer science",
    ]
}
skills_data = {"Skill": ["Math", "Computer Science"]}
courses_df = pd.DataFrame(courses_data)
skills_df = pd.DataFrame(skills_data)

# lotus sem join 
res = courses_df.sem_join(skills_df, "Taking {Course Name} will help me learn {Skill}")
print(res)

Citation

If you use LOTUS or semantic operators in a research paper, please cite this work as follows:

@misc{patel2024lotusenablingsemanticqueries,
      title={LOTUS: Enabling Semantic Queries with LLMs Over Tables of Unstructured and Structured Data},
      author={Liana Patel and Siddharth Jha and Carlos Guestrin and Matei Zaharia},
      year={2024},
      eprint={2407.11418},
      archivePrefix={arXiv},
      primaryClass={cs.DB},
      url={https://arxiv.org/abs/2407.11418},
}

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lotus_ai-0.3.0.tar.gz (32.5 kB view details)

Uploaded Source

Built Distribution

lotus_ai-0.3.0-py3-none-any.whl (43.4 kB view details)

Uploaded Python 3

File details

Details for the file lotus_ai-0.3.0.tar.gz.

File metadata

  • Download URL: lotus_ai-0.3.0.tar.gz
  • Upload date:
  • Size: 32.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.10.15

File hashes

Hashes for lotus_ai-0.3.0.tar.gz
Algorithm Hash digest
SHA256 e30bc8cb1f5107d875a4d1ab47be7fcd9ee55beb94d42690b17685171b3d7a95
MD5 60e4b61f6360ca2e9d8c3a3e14e582a4
BLAKE2b-256 16aead0bc89d699eca9fa4536461eeff7052c6a14c70bc20838be6d1f41da7f9

See more details on using hashes here.

File details

Details for the file lotus_ai-0.3.0-py3-none-any.whl.

File metadata

  • Download URL: lotus_ai-0.3.0-py3-none-any.whl
  • Upload date:
  • Size: 43.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.10.15

File hashes

Hashes for lotus_ai-0.3.0-py3-none-any.whl
Algorithm Hash digest
SHA256 94ae5fe667583c2d4ad3d4b259d7b793c5840e6954c50a1ebb1babf907619ba2
MD5 22c1b3269383fd556b7281752c125689
BLAKE2b-256 4fc77fea130df4e32938f81dd22f7a3221d255c754242d3e53328d898faaf167

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page