pinecone-cli 1.1

pinecone-cli is a command-line client for interacting with the pinecone vector embedding database.

pinecone-cli

Pinecode-cli is a command-line interface for control and data plane interfacing with Pinecone.

In addition to ALL of the Pinecone "actions/verbs", Pinecone-cli has several additional features that make Pinecone even more powerful including:

• Upload vectors from CSV files
• Upload embeddings of text from a given website URL. Embeddings generated by OpenAI embeddings API.
• New "head" command to peak into a given index, similar to "head" in linux/unix.

Install

Feel free to use the tool directly from source here, or just

pip install pinecone-cli

Pypi here: (https://pypi.org/project/pinecone-cli/)

Usage

The CLI depends on a couple of simple environment variables:

• Your Pinecone API Key
• The region/environment of your Pinecone indexes

There is a simple order in which the CLI picks them up

1. .env file in the current working dir.
2. The environment variable your shell is in.
3. Command line arguments overriding the above.

Let's look at a simple .env file:

PINECONE_API_KEY=123456-9876-ABCDEF
PINECONE_ENVIRONMENT=us-west1-gcp

Of course setting in the shell is about the same:

% export PINECONE_API_KEY=1234-4567-abc

Otherwise you wind up having to pass the key as so:

% pinecli query --apikey=1234 ....

The pattern for using the tool is to invoke 'pinecli' and then use a command. The list of commands appears with --help

% pinecli.py --help
Usage: pinecli.py [OPTIONS] COMMAND [ARGS]...

  A command line interface for working with Pinecone.

Options:
  --help  Show this message and exit.

Commands:
  askquestion               Queries Pinecone with a given vector.
  configure-index-pod-type  Configures the given index to have a pod type.
  configure-index-replicas  Configures the number of replicas for a given
                            index.
  create-collection         Creates a Pinecone collection from the argument
                            'source_index'
  create-index              Creates a Pinecone Index.
  delete-all                Delete all vectors (note separate command [delete-
                            index] can completely delete an index)
  delete-collection         Deletes a collection.
  delete-index              Deletes an index.  You will be prompted to
                            confirm.
  describe-collection       Describes a collection.
  describe-index            Describes an index.
  describe-index-stats      Prints out index stats to stdout.
  fetch                     Fetches vectors from Pinecone specified by the
                            vectors' ids.
  head                      Shows a preview of vectors in the
                            <PINECONE_INDEX_NAME>
  list-collections          Lists collections for the given apikey.
  list-indexes              Lists the indexes for your api key.
  minimize-cluster          Minimizes everything for a cluster to lowest
                            settings.
  query                     Queries Pinecone with a given vector.
  update                    Updates the index based on the given id passed in.
  upsert                    Extracts text from url arg, vectorizes w/ openai
                            embedding api, and upserts to Pinecone.
  upsert-file               Upserts a file (csv) into the specified index.
  upsert-random             Upserts a vector(s) with random dimensions into
                            the specified vector.
  upsert-webpage            Extracts text from url arg, vectorizes w/ openai
                            embedding api, and upserts to Pinecone.
  version                   Prints version number.

Commands With Examples

Before you can use Pinecone an index is required. We can now do this on the commandline rather than in the UI: (Note not all of the cmdline options are required, they're shown here to demonstrate functionality and control)

% pinecli create-index myindex --dims=1536 --metric=cosine --pods=2 --replicas=2 --shards=1 --pod-"type=p2.x1"

Note that for any command, if you want an exhasuive description of cmdline options, simply do something similar to the below, where "create-index" is replaced by one of the commands:

% pinecli create-index --help
Usage: pinecli create-index [OPTIONS] PINECONE_INDEX_NAME

  Creates the Pinecone index named <PINECONE_INDEX_NAME>

Options:
  --apikey TEXT             Pinecone API Key
  --region TEXT             Pinecone Index Region
  --dims INTEGER            Number of dimensions for this index  [required]
  --metric TEXT             Distance metric to use.  [required]
  --pods INTEGER            Number of pods  [default: 1]
  --replicas INTEGER        Number of replicas  [default: 1]
  --shards INTEGER          Number of shards  [default: 1]
  --pod-type TEXT           Type of pods to create.  [required]
  --source_collection TEXT  Source collection to create index from
  --help                    Show this message and exit.

Create Index From Collection

You can also create an index from a collection (effectively an index backup) as so:

% pinecli.py create-collection --collection_name='testercollection'  --source_index='mysourcecollection'

Let's try some commands showing two missing features I'd love to have had over the last year: a "head" command and a quick "stats" command:

Index Stats Including Number of Vectors

% pinecli describe-index-stats myindex
Dimensions: 1536
Vectors: 7745
Index_Fullness: 0.0
Namespace data:
        : 7745

Head command to preview vectors

% pinecli head kids-facenet
{'matches': [{'id': 'bubba_50.jpg.vec',
              'metadata': {},
              'score': 12.182938,
              'values': [-0.016061664,
                         -0.4495437,
                         -0.034082577,
                         .....

Now, let's query some nonsensical data from the index named 'upsertfile'

Inserting a vector directly

Note the double quites around the vector

% pinecli query myindex  "[1.2, 1.0, 3.0]" --print-table  --include-meta=True
                      🌲 upsertfile ns=() Index Results                      
┏━━━━━━┳━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃   ID ┃ NS ┃ Values                   ┃                Meta ┃        Score ┃
┡━━━━━━╇━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ vec1 │    │ 0.1,0.2,0.3              │  {'genre': 'drama'} │    0.9640127 │
│ vec2 │    │ 0.2,0.3,0.4              │ {'genre': 'action'} │    0.9552943 │
│  abc │    │ 0.23223,-1.333,0.2222222 │      {'foo': 'bar'} │ -0.083585836 │
│  ghi │    │ 0.23223,-1.333,0.2222222 │      {'bar': 'baz'} │ -0.083585836 │
└──────┴────┴──────────────────────────┴─────────────────────┴──────────────┘

Markdown of course does a great job of mangling great terminal output so here's a screenshot from using ---print-table:

You can of course not output the pretty table by removing --print-table:

% pinecli query myindex "[1.2, 1.0, 3.0]" --include-meta=True
{'matches': [{'id': 'vec1',
              'metadata': {'genre': 'drama'},
              'score': 0.9640127,
              'values': [0.1, 0.2, 0.3]},
              ...

Upsert Vectors in Command Line Manually

Following the Pinecone vector format of the tuple formatted as:

('vectorid', [vecdim1, vecdim2, vecdim3], {'metakey':'metaval'})

You can pass this in as a comma separated list of vectors on the command line:

pinecli upsert myindex "[('vec1', [0.1, 0.2, 0.3], {'genre': 'drama'}), ('vec2', [0.2, 0.3, 0.4], {'foo': 'bar'}),]"

Upsert CSV file

Upserting a csv file is trivial. Simply create your csv file with any headings you have, but there must be at least a labeled id column and a labeled vector column for the vectors. Here's an example of a CSV file that is clearly a DataFrame dump due to the index column on the left which works great w/ pinecone-cli:

index,my_id_column,my_vectors_column,Metadata
1,abc,"[0.23223, -1.333, 0.2222222]",{'foo':'bar'}
2,ghi,"[0.23223, -1.333, 0.2222222]",{'bar':'baz'}

The name of those columns in the header row can be arbitrary or you can name then "id", "vectors" and "metadata" which is our default assumption. If you have custom column names and don't want to change them, just pass in the --colmap argument which takes in a python dictionary mapping "id" and "vectors" to the naming you have in your csv. For example: "{"id":"my_id_column", "vectors":"my_vectors_column"}

Note that as in other CSV file for Dataframes, we need an index column as in the example above. Here's an example using the CSV headers and format above with the correct colmap argument:

% pinecli upsert-file  embeddings.csv myindex "{'id':'my_id_column', 'vectors':'my_vectors_column'}"

More on CSV Formatting

For now you will need to manually provide an index column (we are using dataframes under the hood.)

Upserting Vector Embeddings of Webpage Text

pinecone-cli was built to make using Pinecone extremely easy and fast. We have integrated OpenAI (others coming) - using its embedding APIs to fetch embeddings. We then upload them into your index for you, making uploading embeddings of an entire website's text - trivial.

% pinecli upsert-webpage https://menlovc.com lpfactset  --openaiapikey=12345-9876-abcdef
[nltk_data] Downloading package punkt to /Users/tim/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
100%|████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 61680.94it/s]
['About Us  Our Promise  Focus Areas   Consumer  Cloud Infrastructure  Cybersecurity  Fintech  Healthcare  SaaS  Supply Chain and Automation    Team  Portfolio  Perspective            When we invest, we’re invested. Our promise to founders        Building a business is a team sport. As investors, we don’t just sit on the sidelines but do whatever it takes to help our teams win. About Us    The founders we back don’t limit themselves to what is, but relentlessly pursue what could be. We invest in transformative technology companies that are changing the way we live and work. Portfolio    Menlo Labs starts companies. We work shoulder-to-shoulder 
....
100%|█████████████████████████| 1/1 [00:00<00:00,  1.11it/s]

Upsert Random Vectors

One of the more useful things we can do with pinecone-cli is insert random vectors, primarily for testing. Often we will create our index and the length of the vector will be 1,536 dimensions, for example. Instead of writing a bunch of code to go suddenly create those vectors somehow, we can use pinecone-cli to start generating vectors and upserting them:

% pinecli upsert-random  upsertfile  --num_vector_dims=1536 --num_vectors=10 --debug                                                                                                                                      
upserted_count: 10
1it [00:00,  4.36it/s]  

The example above inserts 10 vectors that each have 1,536 random vectors in them. Note that the id for each vector is simply f'id-{i}' where is is the ith row (vector) inserted.

Query Vectors

Querying can be done in two ways on the cmdline - pass in an actual vector string literal, or ask Pinecone to query randomly (maybe you want to just look at them or look at a TSNE). In the example below, the last argument (required) is either the string 'random' or an actual vector such as '[0.0, 1.0, 3.14569]'. Let's try random:

% pinecli query myindex random

You can also plot a TSNE plot to view clustering of your vectors by using the -show-tsne=True flag. Note that this will pop up the plt plot by default.

% pinecli query lpfactset random --show-tsne=true --topk=2500 --num-clusters=4

Fetching Vectors

Fetching is simple - just pass in the vector id(s) of the vectors you're looking for as a comma separated list:

% pinecli fetch myindex --vector_ids="05b4509ee655aacb10bfbb6ba212c65c,c626975ec096b9108f158a56a59b2fd6"

{'namespace': '',
 'vectors': {'05b4509ee655aacb10bfbb6ba212c65c': {'id': '05b4509ee655aacb10bfbb6ba212c65c',
                                                  'metadata': {'content': 'Chime '
                                                                          'Scholar '
                                                                          'spotlight: '

Updating Vectors

Updating vectors is simple - pass the id of the vector and the updating vector as below:

% pinecli update "id-9" myindex  "[0.0, 1.0, 3.0]"

List Operations

pinecone-cli has all of the necessary 'list' operations as shown below:

List Indexes

This gives you a list of all indexes under your api key:

% pinecli list-indexes                                    
cli
cli2
cli3
drivertest

List Indexes Fully

You can also view the index data similarly to what you'd get on the Pinecone page using the (--print-table) flag with all data such as pods, metric type, shards, etc:

% pinecli.py list-indexes  --print-table      

List Collections

This obviously lists the collections you've created:

% pinecli list-collections                                    
cli
cli2
cli3
drivertest

Other Meta Operations

We showed the "describe-index-stats" command at the top of this page. There is also "describe-index" which provides the following:

% pinecli describe-index lpfactset
Name: lpfactset
Dimensions: 1536
Metric: cosine
Pods: 1
PodType: p2.x1
Shards: 1
Replicas: 1
Ready: True
State: Ready
Metaconfig: None
Sourcecollection: 

Describe Collections

% pinecli describe-collection testcoll
Name: testcoll
Dimensions: 1536
Vectors: 124
Status: Ready
Size: 3917544

Deleting Vectors From Index

This will basically do a rm from the index and clear it out, but will notDELETE* the index. In other words, vector count will be 0.

% pinecli delete-all myindexname

Deleting Indexes

Deleting an index is straightforward. To prevent catastrophic accidents, you'll be prompted to type in the name of the index backwards:

% pinecli delete-index myindex2
Type name of index backwards to confirm: : 2xedniym