llm-app 0.3.3

LLM-App is a library for creating responsive AI applications leveraging OpenAI/Hugging Face APIs to provide responses to user queries based on live data sources. Build your own LLM application in 30 lines of code, no vector database required.

LLM App

Pathway's LLM App is a Python framework to create and deploy pipelines for data ingestion, processing, and retrieval for your AI Application, based on the most up-to-date knowledge available in your data sources.

You can:

• Process streaming data with LLMs, and get realtime updates for your questions. See the alert example.
• Run data transformation pipelines with LLMs. With our unstructured to sql example, you can easily insert data from your PDF documents directly into an SQL database.
• Connect static and dynamic information sources to LLMs and apply custom transformation/decision/filtering processes with natural language.
• Unify back-end, embedding, retrieval, and LLM tech stack into a single application.

Quick links - 👀Why LLM App? 💡Use cases 📚 How it works 🌟 Key Features 🏁 Get Started 🎬 Showcases 🛠️ Troubleshooting 👥 Contributing

Examples

Example	Description
alert	Ask questions, get alerted whenever response changes. Pathway is always listening for changes, whenever new relevant information is added to the stream (local files in this example), LLM decides if there is a substantial difference in response and notifies the user with a Slack message.
drive_alert	Alert example on steroids. Whenever relevant information on Google Docs is modified or added, get real-time alerts via Slack.
contextless	This simple example calls OpenAI ChatGPT API but does not use an index when processing queries. It relies solely on the given user query. We recommend it to start your Pathway LLM journey.
contextful	This default example of the app will index the jsonlines documents located in the data/pathway-docs directory. These indexed documents are then taken into account when processing queries. The pathway pipeline running in this mode is located at examples/pipelines/contextful/app.py.
contextful_s3	This example operates similarly to the contextful mode. The main difference is that the documents are stored and indexed from an S3 bucket, allowing the handling of a larger volume of documents. This can be more suitable for production environments.
unstructured	Process unstructured documents such as PDF, HTML, DOCX, PPTX, and more. Visit unstructured-io for the full list of supported formats.
local	This example runs the application using Huggingface Transformers, which eliminates the need for the data to leave the machine. It provides a convenient way to use state-of-the-art NLP models locally.
unstructuredtosql	This example extracts the data from unstructured files and stores it into a PostgreSQL table. It also transforms the user query into an SQL query which is then executed on the PostgreSQL table.

Quickstart

In Pathway, we can create tables from different stream sources, whether incoming customer e-mails or PDFs on Google Drive. Let us create an example table of applicants and a set of universities. We want GPT to filter out applicants based on their universities and majors. We then set up notifications where we will be kept up to date with the latest applications that meet the criteria.

import pathway as pw
from llm_app.model_wrappers import OpenAIChatGPTModel, OpenAIEmbeddingModel
# we create static table for debugging purpose, in real world, this can be any streaming data connector or any data source such as local folder or S3 bucket
applications_table = pw.debug.parse_to_table("""name degree GPA University
1 Alice Math 3.2 UBC
2 Matthew Linguistics 3.4 CalTech
3 Bob CS 3.25 MIT""")

tracked_universities = pw.debug.parse_to_table("""University
1 Stanford
2 MIT
3 Rice""")

agg_universities = tracked_universities.reduce(universities_list=pw.reducers.tuple(tracked_universities.University))
#  make a tuple with column, ('Stanford', 'MIT', 'Rice')
model = OpenAIChatGPTModel(api_key=api_key)

combined_tables = applications_table.join(agg_universities).select(*pw.left, *pw.right)  # horizontal stack and keep all columns

# we want to filter candidates coming from streaming source, lets create a user defined function with parameters that will be filled on run time
@pw.udf
def create_prompt(user_degree, user_university, tracked_universities):
    prompt = f"""Given list of tracked universities, applicant degree and university, return True if:
    applicant studies in a tracked university towards a software related degree.
    tracked universities: {tracked_universities}
    Applicant degree: {user_degree}
    Applicant university: {user_university}
    Bool:"""
    return prompt

prompt_table = combined_tables.select(prompt=create_prompt(pw.this.degree, pw.this.University , pw.this.universities_list))

import ast
# we will filter based on the value of this column, LLM returns a string ('True' or 'False') and we evaluate it as bool
@pw.udf
def udf_eval(input) -> bool:  #  type hinting allows pathway to pick on type
    return ast.literal_eval(input)

# ask gpt to fill the decision and parse it as bool
response_table = combined_tables + prompt_table.select(
        result=udf_eval(model.apply(
            pw.this.prompt,
            locator="gpt-3.5-turbo",
            temperature=0,
            max_tokens=200,
        )),
    )

notification_table = (response_table.filter(response_table.result)
                      .without('universities_list', 'result'))  # filter rows based on the result decided by GPT, discard some useless columns

# set up alert function that will be run on each update to this table
def send_alert(key, row: dict, time: int, is_addition: bool):
    print("New candidate!")
    if not is_addition:
        return
    print(f"{key}, {row}, {time}, {is_addition}")

# pipe together table and alerting
pw.io.subscribe(notification_table, send_alert)

pw.run()  # run the pipeline
# Out: New candidate!
# Out: 2, Matthew Linguistics 3.4 CalTech, 1699888372, True

Why LLM App?

1. Simplicity - Simplifies your AI pipeline by consolidating capabilities into one platform. No need to integrate and maintain separate modules for your Gen AI app: Vector Databases (e.g. Pinecone/Weaviate/Qdrant) + LangChain + Cache (e.g. Redis) + API Framework (e.g. Fast API).
2. Real-time data syncing - Syncs both structured and unstructured data from diverse sources, enabling real-time Retrieval Augmented Generation (RAG).
3. Easy alert setup - Configure alerts for key business events with simple configurations. Ask a question, get updated when new info is available.
4. Scalability - Handles heavy data loads and usage without degradation in performance. Metrics help track usage and scalability.
5. Monitoring - Provide visibility into model behavior via monitoring, tracing errors, anomaly detection, and replay for debugging. Helps with response quality.
6. Security - Designed for the enterprise with capabilities like Personally Identifiable Information (PII) detection, content moderation, permissions, and version control. Run this in your private cloud with local LLMs.

Use cases

LLM App examples can be used as templates for developing multiple applications running on top of Pathway. Here are examples of possible uses:

• Build your own Discord AI chatbot that answers questions (this is what you see covered in the video!). Or any similar AI chatbot.
• Ask privacy-preserving queries to an LLM using a private knowledge base that is frequently updated.
• Extend Kafka-based streaming architectures with LLMs.
• Process LLM queries in bulk with prompts created automatically out of input data streams.
• Obtain structured data on the fly out of streams of documents.
• Validate incoming documents against existing documents with an LLM.
• Monitor live information streams with an LLM: news and social media, spotting fake news, travel disruptions...

How it works

The default contextful pipeline launches an application which connects to a source folder with documents, stored in AWS S3 or locally on your computer. The app is always in sync with updates to your documents, building in real-time a "vector index" using the Pathway package. It waits for user queries that come as HTTP REST requests, then uses the index to find relevant documents and responds using OpenAI API or Hugging Face in natural language. This way, it provides answers that are always best on the freshest and most accurate real-time data.

The app can also be combined with streams of fresh data, such as news feeds or status reports, either through REST or a technology like Kafka. It can also be combined with extra static data sources and user-specific contexts, for example, to eliminate the ambiguity problems of natural language with clearer prompts and better contexts.

Read more about the implementation details and how to extend this application in our blog article.

Watch it in action

▶️ Building an LLM Application without a vector database - by Jan Chorowski

Features

Key Features

• Extract meaning from raw text - Set up data ingestion pipelines to extract information, entities, and other structured data from raw text.
• Real-time Alerts - Set up custom logic on top of streaming data and get realtime alerts.
• Real-time document indexing pipeline - This pipeline reads data directly from S3-compatible storage, without the need to query an extra vector document database.
• Model testing - Present and past queries can be run against fresh models to evaluate their quality.
• HTTP REST queries - The system is capable of responding in real-time to HTTP REST queries.

Advanced Features

• Local Machine Learning models - LLM App can be configured to run with local LLMs and embedding models, without making API calls outside of the User's Organization.

• Live data sources - The library can be used to handle live data sources (news feeds, APIs, data streams in Kafka), as well as to include user permissions, a data security layer, and an LLMops monitoring layer.

• User session handling - The library's query-building process can be used to handle user sessions.

• To learn more about advanced features see: Features for Organizations.

Coming Soon

• Splitting the application into indexing and request-serving processes easily.
• Expanding context doc selection with a graph walk / support for a HNSW variant.
• Model drift and monitoring setup.
• A guide to model A/B testing.
• OpenAI API observability with Pathway with zero added latency.

Get Started

Prerequisites

1. Make sure that Python 3.10 or above installed on your machine.
2. Download and Install Pip to manage project packages.
3. [Optional if you use OpenAI models]. Create an OpenAI account and generate a new API Key: To access the OpenAI API, you will need to create an API Key. You can do this by logging into the OpenAI website and navigating to the API Key management page.
4. [Important if you use Windows OS]. Example only supports Unix-like systems (such as Linux, macOS, BSD). If you are a Windows user, we highly recommend leveraging Windows Subsystem for Linux (WSL) or Dockerize the app to run as a container.
5. [Optional if you use Docker to run samples]. Download and install docker.

Follow these steps to install and get started with examples. You can also take a look at the application showcases.

Step 1: Clone the repository

This is done with the git clone command followed by the URL of the repository:

git clone https://github.com/pathwaycom/llm-app.git

Next, navigate to the repository:

cd llm-app

Step 2: Set environment variables

Create an .env file in the root directory and add the following environment variables, adjusting their values according to your specific requirements and setup.

Environment Variable	Description
APP_VARIANT	Determines which pipeline to run in your application. Available modes are [contextful, s3, contextless, local, unstructuredtosql]. By default, the mode is set to contextful.
PATHWAY_REST_CONNECTOR_HOST	Specifies the host IP for the REST connector in Pathway. For the dockerized version, set it to 0.0.0.0 Natively, you can use 127.0.0.1
PATHWAY_REST_CONNECTOR_PORT	Specifies the port number on which the REST connector service of the Pathway should listen.Here, it is set to 8080.
OPENAI_API_KEY	The API token for accessing OpenAI services. If you are not running the local version, pleaseremember to replace it with your personal API token, which you can generate from your account on openai.com.
PATHWAY_CACHE_DIR	Specifies the directory where cache is stored. You could use /tmpcache.

For example:

APP_VARIANT=contextful
PATHWAY_REST_CONNECTOR_HOST=0.0.0.0
PATHWAY_REST_CONNECTOR_PORT=8080
OPENAI_API_KEY=<Your Token>
PATHWAY_CACHE_DIR=/tmp/cache

Step 3: Build and run the app

You can install and run the LLM App in two different ways.

Using Docker

Docker is a tool designed to make it easier to create, deploy, and run applications by using containers. Here is how to use Docker to build and run the LLM App:

docker compose run --build --rm -p 8080:8080 llm-app-examples

If you have set a different port in PATHWAY_REST_CONNECTOR_PORT, replace the second 8080 with this port in the command above.

When the process is complete, the App will be up and running inside a Docker container and accessible at 0.0.0.0:8080. From there, you can proceed to the "Usage" section of the documentation for information on how to interact with the application.

Native Approach

• Install poetry:

  pip install poetry
  

• Install llm_app and dependencies:

  poetry install --with examples --extras local
  

  You can omit --extras local part if you're not going to run local example.

• Run the examples: You can start the example with the command:

  poetry run ./run_examples.py contextful
  

Step 4: Start to use it

1. Send REST queries (in a separate terminal window): These are examples of how to interact with the application once it's running. curl is a command-line tool used to send data using various network protocols. Here, it's being used to send HTTP requests to the application.

   curl --data '{"user": "user", "query": "How to connect to Kafka in Pathway?"}' http://localhost:8080/
   
   curl --data '{"user": "user", "query": "How to use LLMs in Pathway?"}' http://localhost:8080/
   

   If you are on windows CMD, then the query would rather look like this

   curl --data "{\"user\": \"user\", \"query\": \"How to use LLMs in Pathway?\"}" http://localhost:8080/
   

2. Test reactivity by adding a new file: This shows how to test the application's ability to react to changes in data by adding a new file and sending a query.

   cp ./data/documents_extra.jsonl ./data/pathway-docs/
   

   Or if using docker compose:

   docker compose exec llm-app-examples mv /app/examples/data/documents_extra.jsonl /app/examples/data/pathway-docs/
   

   Let's query again:

   curl --data '{"user": "user", "query": "How to use LLMs in Pathway?"}' http://localhost:8080/
   

Step 5: Launch the User Interface:

Go to the examples/ui/ directory (or examples/pipelines/unstructured/ui if you are running the unstructured version.) and execute streamlit run server.py. Then, access the URL displayed in the terminal to engage with the LLM App using a chat interface.

Bonus: Build your own Pathway-powered LLM App

Simply add llm-app to your project's dependencies and copy one of the examples to get started!

Showcases

• Python sales - Find real-time sales with AI-powered Python API using ChatGPT and LLM (Large Language Model) App.

• Dropbox Data Observability - See how to get started with chatting with your Dropbox and having data observability.

Troubleshooting

Please check out our Q&A to get solutions for common installation problems and other issues.

Raise an issue

To provide feedback or report a bug, please raise an issue on our issue tracker.

Contributing

Anyone who wishes to contribute to this project, whether documentation, features, bug fixes, code cleanup, testing, or code reviews, is very much encouraged to do so.

To join, just raise your hand on the Pathway Discord server (#get-help) or the GitHub discussion board.

If you are unfamiliar with how to contribute to GitHub projects, here is a Get Started Guide. A full set of contribution guidelines, along with templates, are in progress.

Supported and maintained by

Pathway is a free ultra-performant data processing framework to power your real-time data products and pipelines. To learn more, checkout Pathway's website.