simon-search 0.0.1.post5

A pipeline which allows for the ingestion, storage, and processing of a large body of textual information with LLMs.

#!/Simon

Hello! Welcome to Simon. Simon is a Python library that powers your entire semantic search stack: OCR, ingest, semantic search, extractive question answering, textual recommendation, and AI chat.

Check out this online demo of the tool!

Quick Start

Are you ready to rock 🪨 Simon? Let's do it.

• Pre-reqs
  • PostgreSQL 15 database with the Vector plugin. A service like supabase.io or neon.tech is easiest, but you can also self host.
  • An OpenAI GPT-4 API key
  • Optional: Java, if you want to use Simon's OCR.

After that, we can get started!

Install the Package

You can get the package from PyPi.

pip install simon-search

Connect to Database

import simon

# connect to your database
context = simon.create_context(
  "PROJECT_NAME",               # an arbitrary string id to silo your data.
                                # (store and search are per-project.)
  "sk-YOUR_OPENAI_API_KEY",     # must support GPT-4

  # postgres options. get these from your postgres provider.
  { "host": "YOUR_DB_HOST.com",
    "port": 5432,
    "user": "YOUR_USERNAME",
    "password": "PASSWORD, or None",
    "database": "YOUR_DATABASE_NAME"
  }
)

# if needed, provision the database
simon.setup(context) # do this *only once once per new database*!!

You can also store the OpenAI key and Database info as environment variables or in a .env.

Storing Some Files

ds = simon.Datastore(context)

# storing a remote webpage (or, if Java is installed, a PDF/PNG)
ds.store_remote("https://en.wikipedia.org/wiki/Chicken", title="Chickens")

# storing a local file (or, if Java is installed, a PDF/PNG)
ds.store_file("/Users/test/file.txt", title="Test File")

# storing some text
ds.store_text("Hello, this is the text I'm storing.", "Title of the Text", "{metadata: can go here}")

Search Those Files

We all know why you came here: search!

s = simon.Search(context)

# Semantic Search
results = s.search("chicken habits")

# Recommendation (check out the demo: https://wikisearch.shabang.io/)
results = s.brainstorm("chickens are a species that")

# LLM Answer and Extractive Question-Answering ("Quoting")
results = s.query("what are chickens?")

That's it! Simple as that. Want to learn more? Read the full tutorial to learn about the overall organization of the package.

Friends!

We are always looking for more friends to build together. If you are interested, please reach out by... CONTRIBUTING! Simply open a PR/Issue/Discussion, and we will be in touch.

---

(C) 2023 Shabang Systems, LLC. Built with ❤️ and 🥗 in the SF Bay Area