Pinecone BYOC

Deploy Pinecone in your own cloud account (AWS, GCP, or Azure) with full control over your infrastructure.

Quick Start

Interactive Setup

curl -fsSL https://raw.githubusercontent.com/pinecone-io/pulumi-pinecone-byoc/main/bootstrap.sh | bash

This will:

1. Select your cloud provider (AWS, GCP, or Azure)
2. Check that required tools are installed (Python 3.12+, uv, cloud CLI, Pulumi, kubectl)
3. Verify your cloud credentials
4. Run an interactive setup wizard
5. Generate a complete Pulumi project

Then deploy:

cd pinecone-byoc
pulumi up

Provisioning takes approximately 25-30 minutes.

Prerequisites

Common Tools (Required for All Clouds)

Cloud-Specific Tools

AWS

GCP

Azure

Architecture

┌──────────────────────┐                    ┌───────────────────────────────────────────────┐
│                      │    operations      │         Your AWS/GCP/Azure Account (VPC)      │
│  Pinecone            │───────────────────▶│                                               │
│  Control Plane       │                    │  ┌─────────────┐  ┌─────────────────────────┐ │
│                      │◀───────────────────│  │  Control    │  │                         │ │
│                      │   cluster state    │  │  Plane      │  │    Cluster Manager      │ │
└──────────────────────┘                    │  └─────────────┘  │     (EKS/GKE/AKS)       │ │
                                            │  ┌─────────────┐  └─────────────────────────┘ │
                                            │  │  Heartbeat  │                              │
                                            │  └─────────────┘                              │
┌──────────────────────┐                    │  ┌───────────────────────────────────────────┐│
│                      │◀───────────────────│  │                                           ││
│  Pinecone            │   metrics &        │  │              Data Plane                   ││
│  Observability (DD)  │   traces           │  │                                           ││
│                      │                    │  └───────────────────────────────────────────┘│
└──────────────────────┘                    │  ┌──────────┐  ┌───────────┐  ┌─────────────┐ │
                                            │  │ S3/GCS/  │  |FoundationDB│  │ Route53/    │ │
        No customer data                    │  │ AzureBlob│  │ (in-cluster)|  | CloudDNS/   | │
        leaves the cluster                  │  └──────────┘  └───────────┘  | Azure DNS   | │
                                            │                               └─────────────┘ │
                                            └───────────────────────────────────────────────┘

How It Works

Pinecone BYOC uses a pull-based model for control plane operations:

1. Index Operations - When you create, scale, or delete indexes through the Pinecone API, these operations are queued in Pinecone's control plane
2. Pull & Execute - Components running in your cluster continuously pull pending operations and execute them locally
3. Heartbeat & State - Your cluster pushes health status and state back to Pinecone for monitoring
4. Observability - Metrics and traces (not customer data) are sent to Pinecone's observability platform (Datadog) for operational insights

This architecture ensures:

• Your data never leaves your cloud account - only operational metrics and cluster state are transmitted
• Network security policies remain under your control
• All communication is outbound from your cluster - Pinecone never needs inbound access

Cluster Access

After deployment, configure kubectl:

AWS:

aws eks update-kubeconfig --region <region> --name <cluster-name>

GCP:

gcloud container clusters get-credentials <cluster-name> --region <region> --project <project-id>

Azure:

az aks get-credentials --resource-group <resource-group> --name <cluster-name>

The exact command is output after pulumi up completes.

Upgrades

Pinecone manages upgrades automatically in the background. If you need to trigger an upgrade manually:

pulumi up -c pinecone-version=<new-version>

Replace <new-version> with the target Pinecone version (e.g., main-abc1234).

Configuration

The setup wizard creates a Pulumi stack with these configurable options:

AWS Configuration Options:

GCP Configuration Options:

Azure Configuration Options:

Edit Pulumi.<stack>.yaml to modify these values.

Programmatic Usage

For advanced users who want to integrate into existing infrastructure:

import pulumi
from pulumi_pinecone_byoc.aws import PineconeAWSCluster, PineconeAWSClusterArgs

config = pulumi.Config()

cluster = PineconeAWSCluster(
    "pinecone-aws-cluster",
    PineconeAWSClusterArgs(
        pinecone_api_key=config.require_secret("pinecone_api_key"),
        pinecone_version=config.require("pinecone_version"),
        region=config.require("region"),
        availability_zones=config.require_object("availability_zones"),
        vpc_cidr=config.get("vpc_cidr") or "10.0.0.0/16",
        deletion_protection=config.get_bool("deletion_protection") if config.get_bool("deletion_protection") is not None else True,
        public_access_enabled=config.get_bool("public_access_enabled") if config.get_bool("public_access_enabled") is not None else True,
        tags=config.get_object("tags") or {},
    ),
)

# Export useful values
pulumi.export("environment", cluster.environment.env_name)
pulumi.export("cluster_name", cluster.cell_name)
pulumi.export("kubeconfig", cluster.eks.kubeconfig)

Installation

Install from PyPI with cloud-specific dependencies:

# For AWS
uv add 'pulumi-pinecone-byoc[aws]'

# For GCP
uv add 'pulumi-pinecone-byoc[gcp]'

# For Azure
uv add 'pulumi-pinecone-byoc[azure]'

Troubleshooting

Preflight check failures

The setup wizard runs preflight checks for cloud quotas. If these fail:

AWS:

1. VPC Quota - Request a limit increase via AWS Service Quotas
2. Elastic IPs - Release unused EIPs or request a limit increase
3. NAT Gateways - Request a limit increase
4. EKS Clusters - Request a limit increase

GCP:

1. APIs - Enable required APIs (compute, container, storage, dns)
2. Compute Quotas - Request CPU/disk quota increases via GCP Console
3. GKE Clusters - Request a limit increase if at quota
4. IP Addresses - Release unused static IPs or request more

Azure:

1. Resource Providers - Register required providers (Microsoft.Compute, Microsoft.ContainerService, etc.)
2. vCPU Quotas - Request vCPU quota increases via Azure Portal
3. AKS Clusters - Request a limit increase if at quota
4. Storage Accounts - Ensure unique naming (3-24 lowercase alphanumeric characters)

Deployment failures

If pulumi up fails partway through:

pulumi refresh  # Sync state with actual resources
pulumi up       # Retry deployment

Cluster access issues

Ensure your cloud credentials match the account where the cluster is deployed:

# AWS
aws sts get-caller-identity

# GCP
gcloud auth list
gcloud config get-value project

# Azure
az account show

Cleanup

To destroy all resources:

pulumi destroy

Note: If deletion_protection is enabled (default), you'll need to disable it first or manually delete protected resources.

Developing this project

Unit tests need nothing but the dev group, and are what pytest runs by default:

uv sync --all-extras --group dev
uv run pytest
uv run ruff check . && uv run ruff format --check . && uv run ty check

Anything that provisions real infrastructure is marked and deselected by default. integration builds a network and asserts against it; e2e deploys a whole cluster, takes around an hour, and needs PINECONE_API_KEY:

uv run pytest -m e2e tests/test_vanilla_e2e.py -s

The vanilla e2e is the control run: the module creates its own VPC, so a failure there is a module-wide problem rather than a BYO-VPC one. There are no assertions - a non-zero pulumi up is the failure signal.

Each run writes a redacted log to .e2e-logs/, and leaves the generated project in .e2e/<stack>/, so an interrupted run is torn down with:

cd .e2e/$USER-vanilla-byoc && pulumi destroy --yes

Pass --keep to leave a stack up on success, or --keep-failed to leave it up only when the test fails.

Support

• Documentation
• GitHub Issues