Skip to main content

Pinecone compatiable client for Lantern

Project description

Lantern client compatible with Pinecone API

Install

pip install lantern-pinecone

Sync from Pinecone to Lantern

import lantern_pinecone
from getpass import getpass

lantern_pinecone.init('postgres://postgres@localhost:5432')

pinecone_ids = list(map(lambda x: str(x), range(100000)))

index = lantern_pinecone.create_from_pinecone(
        api_key=getpass("Pinecone API Key"),
        environment="us-east-1-aws",
        index_name="sift100k",
        namespace="",
        pinecone_ids=pinecone_ids,
        recreate=True,
        create_lantern_index=True)

index.describe_index_stats()

index.query(top_k=10, id='45500', namespace="")

NOTE: If you pass create_lantern_index=False only data will be copied under the table of your index name (in this example sift100k) and you can create an index later externally. Without the index most of the index operations will not be accessible via this client.

Extract Metadata Fields

When copying from Pinecone we create a table in this structure: sql (id TEXT, embedding REAL[], metadata jsonb) If you are planning to use the index with raw sql clients, you may want to extract metadata into separate columns, so you could have more complex/nice looking queries over your metadata fields. So if our metadata has this shape { "title": string, "description": string }, we can extract it using this query:

BEGIN;
ALTER TABLE sift100k
ADD COLUMN title TEXT,
ADD COLUMN description TEXT;

-- Update the new columns with data extracted from the JSONB column
UPDATE sift100k
SET
  title = metadata->>'title',
  description = metadata->>'description';


-- Optionally drop the metadata column
ALTER TABLE sift100k DROP COLUMN metadata;

COMMIT;

After doing this your index will most likely be uncomaptible with this python client, and you should use it via raw sql client like psycopg2

Index operations

import os
import lantern_pinecone
import pandas as pd

LANTERN_DB_URL = os.environ.get('LANTERN_DB_URL') or 'postgres://postgres@localhost:5432'
lantern_pinecone.init(LANTERN_DB_URL)

# Giving our index a name
index_name = "hello-lantern"

# Delete the index, if an index of the same name already exists
if index_name in lantern_pinecone.list_indexes():
    lantern_pinecone.delete_index(index_name)


import time

dimensions = 3
lantern_pinecone.create_index(name=index_name, dimension=dimensions, metric="cosine")
index = lantern_pinecone.Index(index_name=index_name)


df = pd.DataFrame(
    data={
        "id": ["A", "B"],
        "vector": [[1., 1., 1.], [1., 2., 3.]]
    })

# Insert vectors
index.upsert(vectors=zip(df.id, df.vector))

index.describe_index_stats()

index.query(
    vector=[2., 2., 2.],
    top_k=5,
    include_values=True) # returns top_k matches


lantern_pinecone.delete_index(index_name)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lantern-pinecone-0.0.3.tar.gz (7.7 kB view details)

Uploaded Source

Built Distribution

lantern_pinecone-0.0.3-py3-none-any.whl (7.9 kB view details)

Uploaded Python 3

File details

Details for the file lantern-pinecone-0.0.3.tar.gz.

File metadata

  • Download URL: lantern-pinecone-0.0.3.tar.gz
  • Upload date:
  • Size: 7.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.6

File hashes

Hashes for lantern-pinecone-0.0.3.tar.gz
Algorithm Hash digest
SHA256 b5a3cc54570d12ad1f67d871402990bc893a4284a14b8ad5598b51224e665b4b
MD5 45d602a7b950a3e8470670c81982285d
BLAKE2b-256 fa54a1a9673cdae259d622b86ddc6f627a160104ed1ee6a424907015d170ff6c

See more details on using hashes here.

File details

Details for the file lantern_pinecone-0.0.3-py3-none-any.whl.

File metadata

File hashes

Hashes for lantern_pinecone-0.0.3-py3-none-any.whl
Algorithm Hash digest
SHA256 a50b198677a892a71a74f15e1537cb0d947b8111cae82ea54c9ae17c146e0d08
MD5 2505ddb1567b1cb6ac901401bac800e0
BLAKE2b-256 6451ff7f8ea2a9608bb492a5486eb108aa7a7bbcc0d87cccb4507f8c656085d9

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page