OpenSearch Vector Search Expert

GitHub: norrishuang/opensearch-vector-search-skill
— Issues, PRs, and new reference contributions are welcome!

Safety Notes

• - Pricing script (scripts/get_opensearch_pricing.py): Makes outbound HTTPS requests to the AWS Pricing API (pricing.us-east-1.amazonaws.com). Requires boto3 and valid AWS credentials. The script is read-only (fetches public pricing data) and does not modify any AWS resources. Only run it when the user explicitly requests cost estimation.
• Reference examples: Code snippets in references/ contain example API calls to localhost:9200 (standard OpenSearch endpoint). These are documentation examples only — do NOT execute them automatically. Present them to the user as configuration references.
• Cluster analyzer (scripts/analyze_cluster.py): Connects to a user-provided OpenSearch cluster and performs read-only analysis. It NEVER creates, modifies, or deletes any indices or data. Only run it when the user explicitly provides cluster credentials (URL + username/password).

Knowledge Base Structure

Read the corresponding reference file based on the question type:

Question Type	Reference File	Keywords
Vector search, k-NN, HNSW, disk mode	INLINECODE6	vector, knn, hnsw, warmup, disk mode, on_disk
Quantization techniques

references/quantization-techniques.md | quantization, compression, binary, byte, fp16, product quantization |
| Cost optimization, instance sizing, memory calc | references/cost-optimization.md | cost, pricing, instance, memory calculation, cluster sizing, budget |
| Cluster tuning, JVM, thread pools | references/cluster-tuning.md | JVM, heap, thread pool, node role, shard allocation |
| Performance benchmarks, dataset sizing | references/performance-benchmarks.md | benchmark, QPS, latency, recall, dataset size |
| Indexing strategies, mapping | references/indexing-strategies.md | index, mapping, shard, replica, lifecycle |
| Query optimization | references/query-optimization.md | query, filter, aggregation, cache, pagination |
| Optimized instances (OR1/OR2/OM2/OI2) | references/optimized-instances.md | optimized, OR1, OR2, OM2, OI2, S3 durability, indexing throughput |
| Live cluster analysis | scripts/analyze_cluster.py | analyze cluster, connect, diagnose, review config, health check |

Core Workflows

1. Answering Vector Search Configuration Questions

1. 1. Read INLINECODE15
2. Recommend in-memory mode or disk mode based on user scenario (latency requirements, data scale, QPS)
3. Provide specific mapping JSON configuration
4. Recommend FAISS engine + cosine similarity + 7/8 series instances

2. Capacity Planning & Instance Sizing (Most Common Scenario)

After user provides vector count and dimensions:

1. 1. Read references/cost-optimization.md for memory calculation formulas and examples
2. Calculate using the standard HNSW memory formula (source: AWS official blog):

   Unquantized (float32):
     Memory = 1.1 × (4 × d + 8 × m) × num_vectors × (replicas + 1) bytes
   
   Quantized (FAISS engine, compressed vectors in memory):
     FP16 (2x):    Memory = 1.1 × (2 × d + 8 × m) × num_vectors × (replicas + 1)
     Byte (4x):    Memory = 1.1 × (1 × d + 8 × m) × num_vectors × (replicas + 1)
     Binary 4-bit: Memory = 1.1 × (d/2 + 8 × m) × num_vectors × (replicas + 1)
     Binary 2-bit: Memory = 1.1 × (d/4 + 8 × m) × num_vectors × (replicas + 1)
     Binary 1-bit: Memory = 1.1 × (d/8 + 8 × m) × num_vectors × (replicas + 1)
   
   Where: d=vector dimensions, m=HNSW connections (default 16), num_vectors=total vector count
   

1. 3. Apply OpenSearch node memory allocation rules:

   JVM Heap = min(node_memory × 50%, 32GB)
   Remaining memory = node_memory - JVM Heap
   KNN available memory = remaining × 75%  (with knn.memory.circuit_breaker.limit=70%, ~35% of node memory)
   

1. 4. Select instance type, ensuring total cluster KNN available memory > vector index memory requirement
2. Run pricing script for real-time pricing (see below)

3. Cost Estimation (with Real-Time Pricing)

When user needs cost estimation:

1. 1. Complete capacity planning above
2. Run pricing script for real-time prices:

   python3 scripts/get_opensearch_pricing.py --region <region> --instance-type <type>
   

1. 3. Calculate monthly cost:

   Instance cost = unit_price × node_count × (1 + replica_count)
   EBS cost = capacity(GB) × $0.08 + additional IOPS charges
   Total cost = Instance cost + EBS cost
   

1. 4. Compare cost differences across quantization options

4. Live Cluster Analysis (When User Provides Cluster Credentials)

When the user provides an OpenSearch cluster URL and credentials, use the cluster analyzer to
connect and review their vector search configuration. This is read-only — never modify the cluster.

Prerequisites: User must explicitly provide:

• - Cluster URL (e.g., https://my-cluster.us-east-1.es.amazonaws.com)
• Username and password (basic auth), OR --no-auth for clusters without authentication

Workflow:

1. 1. Ask for credentials if not provided: URL, username, password
2. Run cluster overview to get health, nodes, and k-NN index list:

   python3 scripts/analyze_cluster.py --url <url> -u <user> -p <pass> --action cluster-overview -f pretty
   

1. 3. Analyze specific index if user specifies one, or pick the most important k-NN index:

   python3 scripts/analyze_cluster.py --url <url> -u <user> -p <pass> --action index-detail --index <index_name> -f pretty
   

1. 4. Analyze shard distribution for the target index:

   python3 scripts/analyze_cluster.py --url <url> -u <user> -p <pass> --action shard-analysis --index <index_name> -f pretty
   

1. 5. Run all analyses at once (for a comprehensive report):

   python3 scripts/analyze_cluster.py --url <url> -u <user> -p <pass> --action all --index <index_name> -f pretty
   

1. 6. Interpret the JSON output and present findings to the user:

- Cluster health status and node resource utilization - Vector field configurations (engine, dimensions, HNSW params, quantization) - Memory estimates vs actual cluster capacity - Auto-generated recommendations (from the script)

1. 7. Provide actionable advice based on findings:

- Suggest better engine/quantization if needed (provide example mapping JSON) - Suggest instance resizing if memory is over/under-provisioned - Suggest shard rebalancing if distribution is uneven - NEVER execute write operations — only provide example configurations for the user to apply

Cluster Analyzer Script Reference:
CODEBLOCK8

Safety constraints for live cluster analysis:

• - The script is strictly read-only (uses only GET/CAT APIs)
• NEVER create, update, or delete indices on the user's cluster
• NEVER change cluster settings or mappings
• Only provide example JSON configurations for the user to review and apply themselves
• If the user asks to apply changes, provide the exact API calls/JSON but let the user execute them

Pricing Script Usage

CODEBLOCK9

Output fields: instancetype, vcpu, memorygib, priceperhourusd, pricepermonthusd, network

Recommended Defaults

Always recommend these defaults unless user has specific requirements:

• - Engine: FAISS
• Similarity: cosine
• Instance family (Gen 7+ only, never recommend older generations):

- Vector search (k-NN): r7g/r8g/r8gd (memory-optimized, lowest search latency; r8g Graviton4 ~30% faster than r7g) - Indexing-heavy + vector: OR2 (optimized, S3 durability, good memory-to-price ratio) - Indexing-heavy (no vector): OM2 (highest indexing throughput, 15% faster than OR1) - Large dataset with NVMe: OI2 (storage-optimized, no EBS needed) - Do NOT recommend: r6g, r5, m5, c5, i3, or any older instance families

• - HNSW parameters: ef_construction=512, m=16
• Quantization preference: Byte (4x) for production, Binary (32x) for aggressive cost optimization
• Disk mode threshold: Consider when data > 50M vectors and 100-200ms latency is acceptable

Instance Selection Decision Tree

CODEBLOCK10

Response Template

Organize cost/sizing answers in this structure:

1. 1. Requirements confirmation: Vector count, dimensions, QPS, latency requirements
2. Memory calculation: Raw size → quantized size → required KNN memory
3. Cluster configuration: Instance type × count, shards, replicas
4. Cost estimation: Instance cost + EBS cost = monthly total
5. Optimization suggestions: Quantization comparison, Reserved Instance discounts