refiner 0.0.6

Refiner is a python package that allows you to store text as vectors in Pinecone and then search for similar text. It uses OpenAI to generate embeddings and then uses Pinecone to store and search for similar text.

AI-Refiner

The Refiner Python package can be used to convert and store text and metadata as vector embeddings. Embeddings are generated using OpenAI and stored as vectors in Pinecone. Stored embeddings can then be "queried" using the search method. Matched embeddings contain contextually relevant metadata that can be used for AI chatbots, and semantic search APIs, etc.

Installation

pip install refiner

OpenAI and Pinecone API Keys.

You'll need API keys for OpenAI and Pinecone.

Once you have your API keys, you can either set local ENV variables in a shell:

export PINECONE_API_KEY="API_KEY"
export PINECONE_ENVIRONMENT_NAME="ENV_NAME"
export OPENAI_API_KEY="API_KEY"

or you can create a .env (dotenv) config file and pass in the file path when initializing the Embeddings class:

from refiner.embeddings import Embeddings
embeddings_client = Embeddings(config_file="/path/to/.env")

Your .env file should follow key/value format:

PINECONE_API_KEY="API_KEY"
PINECONE_ENVIRONMENT_NAME="ENV_NAME"
OPENAI_API_KEY="API_KEY"

Create Embeddings

from refiner.embeddings import Embeddings
embeddings_client = Embeddings(config_file="/path/to/.env")
payload = {
    "id": "index-id,
    "text": "Example text to embed",
    "metadata": {"key": "value"}
}
embeddings_client.create(payload, "example-index")
# {'upserted_count': 1}

Semantic Search

embeddings_client = Embeddings(config_file="/path/to/.env")
limit = 10
embeddings_client.search('search text', 'index-id', limit)
# {'matches': [...]}

CLI

You can install the CLI to create and search your vectors.

pip install refiner-cli

The --help option can be used to learn about the create and search commands.

refiner --help
refiner create --help
refiner search --help