qdrant-tools 0.0.2

No project description provided

Migrating from Pinecone to Qdrant Vector Database: A Comprehensive Guide

Installation

pip install qdrant-tools

Usage

from qdrant_tools.vectordb import PineconeExport, QdrantImport, QdrantMode
index_name = "hindi-search" # Existing Pinecone index name

# Init Pinecone

pinecone_export = PineconeExport(index_name=index_name)
print(pinecone_export.index.describe_index_stats())
source_index = pinecone_export.index

# Fetch all vector ids from Pinecone

vector_ids = ["1", "2", "3", "4", "5"] # Example vector ids, list of strings -- this is the main input for the replication
points = pinecone_export.fetch_vectors(vector_ids) # Fetch vectors from Pinecone

# Init Qdrant

vector_dimension = source_index.describe_index_stats()[
"dimension"
] # Get dimension from existing index
qdrant = QdrantImport(mode=QdrantMode.local)
qdrant.create_collection(index_name, vector_dimension)
qdrant.upsert_vectors(index_name, vector_ids, source_index) # source_index is a pinecone index object


# Testing if it works!
qdrant.qdrant_client.search(
    collection_name=index_name,
    query_vector= # query_vector
)

Introduction

Are you considering a transition from Pinecone to Qdrant? If so, this article will guide you through the process, outlining the similarities and differences between the two systems, and providing a step-by-step migration plan.

Understanding the Terminology

Before diving into the migration process, it's important to familiarize yourself with some key terms.

Pinecone Terminology

• Collection: A collection in Pinecone is a static snapshot of the index, signifying a unique set of vectors with attached metadata. Qdrant also has a similar concept, referred to as a collection.
• Index: An index in Pinecone is the principal organizational unit for vector data. It receives and stores vectors, provides query services, and performs various vector operations. It's similar to the concept of an index in Qdrant.
• Pods: Pinecone uses pods, which are pre-configured hardware units, to host and execute its services. More pods typically result in greater storage capacity, reduced latency, and improved throughput. The Qdrant equivalent is yet to be determined.

Qdrant Terminology

• Collection: A collection in Qdrant is a named set of points, where each point is a vector with an associated payload. The concept is similar to a collection in Pinecone.
• Payload: Qdrant allows additional information to be stored with vectors, referred to as a payload. This corresponds to the concept of associated metadata in Pinecone.
• Points: A point in Qdrant is a record composed of a vector and an optional payload, which closely corresponds to the concept of vectors in Pinecone.

To help visualize these concepts, here is a comparison table:

Vector Database	Pinecone	Qdrant
DB Capacity/Perf	Pod	Cluster
Collection	Snapshot of the Index	Named set of points
Index	Index	Collection
Vector	Vector	Point
Metadata	Metadata	Payload
Namespace	One namespace per vector	None. However, indexing is possible
Size	40KB metadata limit	No default. Size can be set during collection creation/updation for vectors. No payload limit

Planning Your Migration

Migrating from Pinecone to Qdrant involves a series of well-planned steps to ensure that the transition is smooth and disruption-free. Here is a suggested migration plan:

1. Understanding Qdrant (1 week): It's important to first get a solid grasp of Qdrant, its functions, and its APIs. Take time to understand how to establish collections, add points, and query these collections.

2. Migration strategy (2 weeks): Create a comprehensive migration strategy, incorporating data migration (copying your vectors and associated metadata from Pinecone to Qdrant), feature migration (verifying the availability and setting up of features currently in use with Pinecone in Qdrant), and a contingency plan (should there be any unexpected issues).

3. Establishing a parallel Qdrant system (1 week): Set up a Qdrant system to run concurrently with your current Pinecone system. This step will let you begin testing Qdrant without disturbing your ongoing operations on Pinecone.

4. Data migration (2-3 weeks): Shift your vectors and metadata from Pinecone to Qdrant. The timeline for this step could vary, depending on the size of your data and Pinecone API's rate limitations.

5. Testing and transition (2 weeks): Following the data migration, thoroughly test the Qdrant system. Once you're assured of the Qdrant system's stability and performance, you can make the switch.

6. Monitoring and fine-tuning (ongoing): After transitioning to Qdrant, maintain a close watch on its performance. It's key to continue refining the system for optimal results as needed.

Bear in mind, these are just rough timelines, and the actual time taken can vary based on the specifics of your setup and the complexity of the migration.

Streamlining Data and Embedding Migration

Copying vectors and metadata from your existing Pinecone index is achievable with the fetch operation in Pinecone. This operation retrieves vectors by their ID from the index, bringing along their vector data and/or metadata. Typically, the fetch latency falls below 5ms (source).

On the other hand, Qdrant organizes its data into collections. Each collection is a named set of points - vectors coupled with a payload - which can be queried. The vectors in a collection share the same dimensionality and are compared using a single metric (source).

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams

client = QdrantClient(host="localhost", port=6333)
client.recreate_collection(
    collection_name="my_collection",
    vectors_config=VectorParams(size=100, distance=Distance.COSINE),
)

The migration from Pinecone to Qdrant involves a two-step process of exporting data from Pinecone and subsequently importing that data into Qdrant.

Navigating Pinecone Export Restrictions

Data extraction from Pinecone must adhere to certain restrictions:

1. Queries: For the parameter top_k, denoting the number of results to return, the maximum value is 10,000. However, if the options include_metadata=True or include_data=True are utilized, the top_k value reduces to a maximum of 1,000.

2. Fetch and Delete: The ceiling for the number of vectors per fetch or delete request is 1,000.

3. Metadata: The upper limit for metadata size per vector is 40 KB. Pinecone does not support null metadata values, and metadata exhibiting high cardinality may cause the pods to reach capacity.

Onboarding Data into Qdrant

In Qdrant, data is expected in float format. The process of importing data into Qdrant involves the creation of a collection and then indexing the data, described in the steps below:

1. Creating a Collection: In Qdrant, a collection is a named set of points (vectors with a payload) among which you can search. All vectors within a collection should share the same dimensionality and be comparable by a single metric.

You can instantiate a collection by sending a PUT request to /collections/{collection_name}, furnished with necessary parameters including the collection name and the dimensions of vectors.

2. Indexing Data: Once the collection is established, data can be indexed into it. This process can be streamlined by implementing a class like SearchClient, that encompasses methods for data conversion, indexing, and searching. The index method within this class should prepare data in the required format and employ the Qdrant client's upsert function to index the data.

Testing your applications post-migration

Post-migration, it is important to validate your applications with both Pinecone and Qdrant.

For Qdrant, the following actions can help test your applications:

1. Creating a Collection: This can be achieved using the recreate_collection method, which sets up a collection with specified parameters such as vector size and distance metric.

2. Inserting Vectors: Vectors can be inserted into your collection using the upsert method. Along with the vectors, you can also add associated metadata, referred to as payload in Qdrant.

3. Querying Vectors: The search method can be used to find similar vectors within the collection. Qdrant also supports advanced queries

Remember, these are ideas that you can start with. Your actual testing scope may be larger based on the specifics of your application.

Performance Considerations

Cost and Network Latency

One of the primary considerations when choosing a vector database system is cost and network latency.

Pinecone recommends operating within known limits and deploying your application in the same region as your Pinecone service for optimal performance. For users on the Free plan, Pinecone runs in GCP US-West (Oregon).

On the other hand, Qdrant is an open-source vector database that can be self-hosted, providing more flexibility. It can be deployed in your own environment, which can help optimize network latency and potentially lower costs.

Throughput and Speed

Pinecone suggests increasing the number of replicas for your index to increase throughput (QPS). It also provides a gRPC client that can offer higher upsert speeds for multi-pod indexes. However, be aware that reads and writes cannot be performed in parallel in Pinecone, which means that writing in large batches might affect query latency

Qdrant, on the other hand, offers different performance profiles 2 based on your specific use case:

1. Low memory footprint with high speed search: Qdrant achieves this by keeping vectors on disk and minimizing the number of disk reads. You can make this even better by configuring in-memory quantization, with on-disk original vectors.

Vector quantization can be used to achieve this by converting vectors into a more compact representation, which can be stored in memory and used for search.

Below is an example of how to configure this in Python:

from qdrant_client import QdrantClient
from qdrant_client.http import models

client = QdrantClient("localhost", port=6333)

client.recreate_collection(
    collection_name="{collection_name}",
    vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
    optimizers_config=models.OptimizersConfigDiff(memmap_threshold=20000),
    quantization_config=models.ScalarQuantization(
        scalar=models.ScalarQuantizationConfig(
            type=models.ScalarType.INT8,
            always_ram=True,
        ),
    ),
)

2. High precision with low memory footprint: For scenarios where high precision is required but RAM is limited, you can enable on-disk vectors and HNSW index. Here is an example of how to configure this in Python:

from qdrant_client import QdrantClient, models

client = QdrantClient("localhost", port=6333)

client.recreate_collection(
    collection_name="{collection_name}",
    vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
    optimizers_config=models.OptimizersConfigDiff(memmap_threshold=20000),
    hnsw_config=models.HnswConfigDiff(on_disk=True)
)

3. High precision with high speed search: If you want high speed and high precision search, Qdrant can achieve this by keeping as much data in RAM as possible and applying quantization with re-scoring. Here is an example of how to configure this in Python:

from qdrant_client import QdrantClient
from qdrant_client.http import models

client = QdrantClient("localhost", port=6333)

client.recreate_collection(
    collection_name="{collection_name}",
    vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
    optimizers_config=models.OptimizersConfigDiff(memmap_threshold=20000),
    quantization_config=models.ScalarQuantization(
        scalar=models.ScalarQuantizationConfig(
            type=models.ScalarType.INT8,
            always_ram=True,
        ),
    ),
)

There are also search/query time changes you can make to influence Qdrant's performance characteristics:

from qdrant_client import QdrantClient, models

client = QdrantClient("localhost", port=6333)

client.search(
    collection_name="{collection_name}",
    search_params=models.SearchParams(
        hnsw_ef=128,
        exact=False
    ),
    query_vector=[0.2, 0.1, 0.9, 0.7],
    limit=3,
)

Overall, Qdrant's flexibility allows for a wide range of performance tuning options to suit various use cases, which could make it a better option for those looking for a customizable vector database.

Please note that when tuning for performance, you must consider your specific use case, infrastructure, and workload characteristics. The suggested configurations are starting points and may need to be adjusted based on actual performance observations and requirements.

Pinecone

Here are some tips for getting the best performance out of Pinecone.

Security Considerations

Qdrant

Qdrant takes a unique approach to security that is minimalist yet flexible, allowing for robust security configurations tailored to each unique deployment environment:

1. Environment-level Security: Qdrant is designed to be deployed in secure environments, such as Kubernetes with Network Policies or a Virtual Private Cloud (VPC). This approach puts the control of security aspects in the hands of the deployer, allowing for custom security measures tailored to the specific needs of the deployment environment. If you're using Qdrant Cloud, it also uses API keys for authentication. Ensure these keys are securely managed.

2. Data Encryption: While Qdrant does not support built-in data encryption, it leaves the choice of encryption strategy to the underlying storage service. This allows for a wide variety of encryption methods to be employed depending on the specific storage solution used, providing greater flexibility.

3. Authentication: Qdrant's can be easily integrated with any existing authentication system at the service level, such as a reverse proxy. This allows for seamless integration with existing infrastructure without the need for modifications to accommodate a built-in authentication system.

4. Network Security: Qdrant leaves network security to be handled at the environment level. This approach allows for a wide range of network security configurations to be employed depending on the specific needs of the deployment environment.

5. Reporting Security Issues: Qdrant encourages the reporting of any security-related issues to their dedicated security email, demonstrating their commitment to ongoing security improvements.

Pinecone

1. Authentication: Pinecone requires a valid API key and environment pair for accessing its services, which can be a limiting factor if integration with an existing authentication system is desired.

2. Data and Network Safeguards: Pinecone runs on a fully managed and secure AWS infrastructure, which may not provide the flexibility required for some deployment scenarios.

3. Compliance and Certifications: While Pinecone's SOC2 Type II certification and GDPR-readiness are reassuring, they may not be sufficient for some organizations who want to work strictly within their VPC. This means deploying an on-premise of Pinecone under enterprise offering, which can be prohibitively expensive for some organizations.ß

4. Security Policies and Practices: Pinecone's rigorous security policies may not align with the security practices of all organizations. This also moves the burden of finding the difference between the security policies and ironing them out to the end user.

5. Incident Management and Monitoring: Pinecone's incident management and monitoring practices are not integrated with the organisation's existing incident management and monitoring systems, potentially complicating incident response.

In conclusion, Qdrant's minimalist approach to security allows for greater flexibility and customization according to the specific needs of the deployment environment. It puts the control of security measures in the hands of the deployer, allowing for robust, tailored security configurations. On the other hand, Pinecone's built-in security measures and compliance certifications provide a comprehensive, ready-to-use security solution that may not provide the same level of customization as Qdrant. The choice between the two depends largely on the specific security needs of your application and the flexibility of your deployment environment.