Pod-Based Indexing in Pinecone

Details of pod-based indexing in Pinecone, along with some example use cases.

Pod-Based Indexing in Pinecone

Pod Types and Sizes:

Pinecone offers different pod types, each optimized for specific use cases:

s1 (Storage-optimized): Suitable for scenarios where storage capacity is critical.

p1 (Performance-optimized): Balances storage and query performance.

p2 (High throughput): Designed for applications requiring minimal latency and high throughput.

You can choose the appropriate pod type based on your requirements.

The default pod size is x1.

After index creation, you can increase the pod size without downtime. Reads and writes continue uninterrupted during scaling.

Resizing completes in about 10 minutes.

Example Python code to change the pod size of an existing index:

from pinecone import Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

pc.configure_index("example-index", pod_type="s1.x2")

Checking Pod Size Change Status:

To monitor the status of a pod size change, use the describe_index operation.

The status field indicates whether the resizing process is ongoing or complete.

Example Python code to check the status:

from pinecone import Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

pc.describe_index("example-index")

Adding Replicas:

Increasing the number of replicas improves throughput (QPS).

All pod-based indexes start with replicas=1.

Example Python code to set the number of replicas for an index:

from pinecone import Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

pc.configure_index("example-index", replicas=4)

Selective Metadata Indexing:

Pinecone indexes all metadata fields by default.

For fast operations on subsets of records, use ID prefixes.

Example Python code to create a pod-based index that only indexes the genre metadata field:

from pinecone import Pinecone

pc = Pinecone(api_key="YOUR_API_KEY")

pc.create_index(name="genre-index", dimension=128, metric="cosine", metadata_fields=["genre"])

Example Use Cases

Semantic Search of News Articles:

Imagine building a semantic search engine for news articles.

You can create an index with relevant metadata fields (e.g., title, content, category).

Users can search for articles related to specific topics or keywords.

Optimize pod type and size based on query latency and throughput requirements.

Movie Recommendations:

For a video streaming application, use a p2 pod to recommend movies based on user preferences.

High throughput is crucial to handle personalized recommendations for a large user base.