Generated banner for Amazon Opensearch Service
Generated logo for Amazon Opensearch Service

Amazon Opensearch Service

AWSOfficialPublished by AWS

  • Skill
  • Workflow
  • Cloud and DevOps
  • Apache-2.0
  • Low risk

From the author

Guides migration, provisioning, search, log-analytics, trace-analytics, and Agentic AI Assistant workflows for Amazon OpenSearch Service and Serverless across six capabilities — migration (Solr/ES/self-managed into AOS/AOSS, schema/query translation, sizing, cutover); provisioning (domain + AOSS lifecycle, upgrades, FGAC, monitoring); search (vector / semantic / hybrid / RAG with Bedrock); log-analytics (PPL, OSI, anomaly detection, Dashboards); trace-analytics (OTel spans, service maps, Data Prepper); ai-assistant (natural language data exploration, incident investigation, root cause analysis). Triggers on OpenSearch, AOS, AOSS, Elasticsearch, Solr, vector/k-NN/semantic/hybrid search, RAG, log analytics, PPL, trace analytics, ISM, FAISS, HNSW, Migration Assistant, UltraWarm, OR1, query my data, analyze logs, investigate errors, root cause analysis.
— the description by AWS

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a unified framework for interacting with Amazon OpenSearch Service (AOS) and OpenSearch Serverless (AOSS). It covers the full lifecycle of OpenSearch operations, including migration from legacy systems like Solr or Elasticsearch, infrastructure provisioning, advanced search implementation (including vector and RAG), and observability through log and trace analytics. It also features an Agentic AI Assistant for natural language data exploration and incident investigation, ensuring users can navigate complex cluster management and data analysis tasks without manual query crafting.

What it does

  • Guides migration workflows from Solr, Elasticsearch, or self-managed OpenSearch to AOS/AOSS.
  • Assists with provisioning, lifecycle management, and monitoring of domains and collections.
  • Facilitates implementation of vector, semantic, and hybrid search, including RAG with Bedrock.
  • Supports log and trace analytics using PPL, OpenTelemetry, and anomaly detection.
  • Provides an Agentic AI Assistant for natural language querying and automated incident root cause analysis.
  • Enforces operational best practices for sharding, storage tiering, and security configurations.

When to use it

  • When planning a migration from legacy search engines to Amazon OpenSearch Service.
  • When setting up or optimizing search capabilities, including vector search and RAG.
  • When troubleshooting cluster health, performance, or shard allocation issues.
  • When performing log or trace analysis to investigate system incidents.
  • When you need to generate optimized PPL or DSL queries using natural language.

Advantages

  • Centralizes complex operational knowledge across six distinct OpenSearch capabilities.
  • Reduces manual query complexity by leveraging the Agentic AI Assistant.
  • Provides clear, actionable guidance on production failure modes and migration blockers.
  • Standardizes interactions through consistent CLI and data-plane patterns.
  • Promotes architectural best practices for scaling and cost-effective storage tiering.

Usage examples

“How do I migrate my existing Solr index to Amazon OpenSearch Service?”
The agent identifies the migration capability, explains that Solr to OpenSearch is a refactor migration, and guides you through using the Migration Assistant for data backfill.
“Why is my cluster status yellow and how can I fix it?”
The agent uses the AI Assistant capability to analyze cluster health, check shard allocation, and provide specific commands or steps to resolve unassigned shards.
“Help me set up a RAG pipeline using Bedrock and my OpenSearch domain.”
The agent routes to the search capability, providing instructions on configuring Bedrock connectors and vector index mappings for semantic retrieval.
“Find the root cause of the 503 errors in my application logs from the last hour.”
The agent utilizes the AI Assistant to query logs, perform time-series analysis, and correlate findings to identify the likely root cause of the errors.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/aws/amazon-opensearch-service/download
  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip amazon-opensearch-service.zip -d .agents/skills

Documentation by AWS

Amazon OpenSearch Service — the unified skill

This skill answers anything about Amazon OpenSearch Service or Serverless across six capabilities. Step 0 below routes the question to ONE capability and points at that capability's entry-point reference. Everything else — when to dispatch, sub-references, capability-specific facts, cross-capability links — lives in the entry-point reference for that capability.

AWS MCP server is recommended, not required. Capability references show standard AWS CLI commands as the primary syntax (e.g., aws opensearch describe-domain, aws opensearchserverless create-collection). Where the AWS MCP server is available, its call_aws tool offers a streamlined alternative — but every operation in this skill MUST work via the AWS CLI alone. Data-plane HTTP calls against AOS / AOSS use awscurl for SigV4-signed requests; this works in both contexts.

Step 0: detect the capability — first thing you do

Pick one of the six capabilities below. State the detected capability in your first sentence (e.g., "Detected capability: SEARCH — semantic search setup with Bedrock embeddings."). Then load the entry-point reference; that file describes when to dispatch, indexes the rest of the capability's files, and routes you to the next step.

CapabilityEntry-point reference
migration — Solr / Elasticsearch / self-managed OpenSearch into AOS or AOSS. Schema/query translation, sizing, cutover.references/assessment-workflow.md
provisioning — Provisioning and managing AOS domains and AOSS collections. Lifecycle, upgrades, storage tiers, FGAC, monitoring.references/provisioning-reference.md
search — Vector / semantic / hybrid / sparse / dense / RAG retrieval. Bedrock connectors, FAISS HNSW vs Lucene.references/search-semantic-search-guide.md
log-analytics — Log search, observability, PPL, OSI ingestion, anomaly detection, OpenSearch Dashboards. Splunk/Datadog/ELK alternatives.references/log-analytics-guide.md
trace-analytics — Distributed traces with OpenTelemetry. Span queries, service maps, Data Prepper.references/trace-analytics-trace-queries.md
ai-assistant — Agentic AI Assistant: auto-discovers indices, generates optimized PPL/DSL queries, summarizes results, and investigates incidents end-to-end. No manual query crafting needed.references/ai-assistant.md

If a prompt spans capabilities (e.g., "migrate from Solr AND set up RAG on the new domain"), pick the dominant capability for the response and close with a one-line handoff to the other capability's entry-point ref.

Universal rules (apply to ALL capabilities)

These rules apply to every response, regardless of capability. Capability-specific rules (sizing math, shape detection, Migration Assistant for Amazon OpenSearch Service capability matrix, k-NN engine selection) live in the entry-point references, not here.

  • Report header (every multi-section response). Begin every multi-section response with a single fenced metadata block: > Generated: <ISO 8601 timestamp> | Skill: amazon-opensearch-service v<N>. Get the time by calling the current_time tool (returns ISO 8601 in UTC). Read the skill version from this file's frontmatter version: field. For one-line answers (terse FOCUSED_OPERATIONAL replies, anti-pattern refusals) the header is optional; for any multi-section deliverable it is REQUIRED. Place it immediately after the report title and before the first ## heading.

  • No dollar estimates (HARD CONSTRAINT). Never produce $X/month, ~$1,500, or any dollar figure. Route every cost question to https://calculator.aws and stop. If a sub-reference contains dollar figures, treat them as informational context only and do NOT pass them through to the user.

  • No credential leakage (HARD CONSTRAINT). Never include master usernames, KMS key ARNs, VPC endpoint URLs, instance IPs, or account IDs in generated output.

  • Pick one for every A-vs-B decision. Name a primary recommendation in one line with a one-sentence reason. A "go with B if..." caveat is allowed AFTER the primary; never lead with conditional-only guidance.

  • Source restatement. The first 2–3 sentences must restate the source (engine + version + scale) when known, or restate the customer's question in concrete terms. The very first text the user sees must NOT be tool narration, meta-commentary, the report title, or simply restating the question verbatim.

  • No marketing tone. Do NOT use "seamless", "robust", "best-in-class", "production-hardened", "enterprise-grade", "world-class", "cleanly", "elegant". Do NOT stack 3+ vague hedges ("typically", "generally", "usually", "in most cases") in a single recommendation — be specific about when it does and does not apply.

  • Cross-capability handoff. When a user prompt spans capabilities (e.g., "migrate from Solr AND set up RAG on the new domain"), pick the dominant capability for the response, then close with a one-line handoff: "For <other capability>, see references/<other-capability>-<entry>.md."

Cross-cutting references (used across multiple capabilities)

These references are not capability-prefixed because they apply across capabilities. Capability entry-point references load them when relevant; SKILL.md never loads them directly.

Assets (assets/): report templates for FULL_ASSESSMENT renderings (Solr-source, ES-source, executive summary).

What this skill does NOT do

  • Estimate dollar costs. Pricing changes monthly and account-specific (RI, Savings Plan, EDP) discount math is outside this skill's reliable scope. Use https://calculator.aws.

  • Move data. Use Migration Assistant for Amazon OpenSearch Service (Historical Data Migration for backfill, Live Traffic Migration for live cutover).

  • Build embedding models. Use Amazon Bedrock or SageMaker.

  • Replace Splunk SPL or Datadog APM 1:1. Some queries / detectors / dashboards need rewriting.

  • Tune relevance for a specific catalog. Use OpenSearch Benchmark big5 workload + your own judgment list.

Guardrail — where this skill's own files live (MCP vs local install)

This skill can be loaded two ways, and they resolve the skill's own bundled files from different places. Determine how the skill was loaded before reading a reference or running a script:

  • Loaded through the AWS MCP server's retrieve_skill tool: The skill is not installed on the local filesystem. You MUST fetch each reference or script via retrieve_skill with the file parameter (e.g. file="references/architecture.md" or file="scripts/deploy.py"), and run the script from the returned content. Do NOT file_read these paths locally — they do not exist on disk.

  • Installed locally (e.g. .kiro/skills/your-skill/ or ~/.claude/skills/your-skill/): Read and run files from the local skill directory using relative paths.

This distinction applies only to the skill's own packaged files. User data and session artifacts are always read from and written to the user's working directory. Never fetch or write customer data through retrieve_skill.

Files

references/sizing.md

Sizing — full math, instance families, and operational thresholds

The summary version (default starting point + key knobs) is in SKILL.md. This file owns the full formulas, instance-family details, JVM/heap mechanics, k-NN memory math, OCU model, and edge-case tuning.

Storage formula

min_storage = source_data × (1 + replicas) × (1 + indexing_overhead) / (1 - linux_reserved) / (1 - aos_overhead)

Defaults (from AWS bp-storage.html):

  • linux_reserved = 0.05 (Linux reserves 5% of file system for root)

  • aos_overhead = 0.20 capped at 20 GiB/instance (AOS reserves 20% up to 20 GiB)

  • indexing_overhead ≈ 0.10 (the index up to 10% of source data)

Simplified rule: min_storage ≈ source_data × (1 + replicas) × 1.45.

For >1 PB workloads, see petabyte-scale.html: 100 GiB shards on OR1.16xlarge.search / i3.16xlarge.search.

Shard math

Source: bp-sharding.html and bp.html.

WorkloadTarget shard size
Search workloads10–30 GiB
Logs / write-heavy30–50 GiB
Petabyte-scale on i3.16xl / OR1up to 100 GiB

Formulas:

  • primary_shards = (source + room_to_grow) × 1.1 / desired_shard_size, rounded up to multiple of data-node count

  • shards_per_node ≤ 25 × GiB_heap — e.g., 32 GiB heap = max 800 shards/node

  • shard_to_cpu ≈ 1.5 vCPU / shard (initial scale point)

Per-node shard cap evolution:

  • ES 7.x and OS ≤ 2.15: 1000 shards/node

  • OS ≥ 2.17: 1000 shards per 16 GiB JVM heap, up to 4000 shards/node max

  • Multi-AZ-with-Standby: 1000 shards/node always (regardless of OS version)

  • Cluster-wide cap (Multi-AZ-with-Standby): 75,000 shards total

JVM heap

RuleValueSource
Heap size50% of RAM, capped at 32 GiBauto-tune.html, cloudwatch-alarms.html
Customer-tunable?NO — set automatically per instance classAWS doc
Compressed-oops ceiling32 GiB JVM limitJVM behavior
Pressure write-block triggerJVMMemoryPressure > 92% for 30 minhandling-errors.html
Pressure write-block releaseJVMMemoryPressure ≤ 88% for 5 minhandling-errors.html
Steady-state target< 80%bp.html

Why 32 GiB ceiling: Above ~32 GiB, JVM disables compressed object pointers (compressed oops), and pointer overhead doubles, eroding any RAM gains.

Beyond 32 GiB RAM: scale horizontally (more nodes), not vertically. The service supports up to 64 GiB RAM single-instance, then enforces horizontal scaling.

Operational thresholds

  • Refresh interval: default 1s. Recommend 30s+ for write-heavy workloads. (bp.html)

  • Bulk request size: 3–5 MiB starting point. (bp.html)

  • Disk watermarks: 85% / 90% / 95% (low / high / flood) — defaults per Elasticsearch / OpenSearch; index goes read-only at flood. See gotcha #18 for the read-only-block consequence and recovery.

    • More granular: cluster blocks writes when free storage drops below 20% OR 20 GiB (whichever is greater).

  • EBS burst balance: notification when GP2 burst < 70%, follow-up at < 20%.

  • UltraWarm cost-effective threshold: ~2.5 TiB hot data. (bp.html)

  • Snapshot retention: AOS automated snapshots kept 14 days (hourly, up to 336). Manual snapshots bill against your S3 bucket at standard rates plus PUT costs.

Topology defaults

Terminology: this skill uses cluster manager (the modern OpenSearch name; formerly "master node" in pre-2.x ES / OS). AWS APIs and CLI flags retain the legacy spelling — e.g., --dedicated-master-enabled, DedicatedMasterCount in aws opensearch create-domain — and are quoted verbatim where they appear. Prose uses "cluster manager".

  • Cluster managers: exactly 3 dedicated, in 3 AZs. Quorum requires odd count; 3 is the minimum that survives single-node failure. NEVER use 1, 2, 4, or 5.

  • Cluster manager sizing (OS 2.17+):

    • 8 GiB cluster manager → up to 30 nodes / 15K shards

    • 32 GiB cluster manager → up to 120 nodes / 60K shards

    • 256 GiB cluster manager → up to 1002 nodes / 500K shards

  • Cluster managers required when ≥ 3 data nodes OR ≥ 10 indexes.

  • Data nodes: ≥ 2 minimum. Multi-AZ-with-Standby uses multiples of 3, with 2 replicas.

  • AZs: 3 for prod (Multi-AZ; Multi-AZ-with-Standby is "available at no extra cost").

  • Replicas: 1 default; 2 for high-availability search workloads; 0 only for ephemeral logs.

Instance family selection (current generation)

Default rule: Graviton r-family (r7g/r8g) for memory-bound search, m-family (m7g/m8g) for cluster managers; OR1/OR2 for write-heavy logs only (write-once read-rare profile). Pick previous-gen (r6g/r6gd) only with explicit justification — existing RIs, specific compatibility need.

For the current list of supported instance types, EBS+Instance-Store profiles, regional availability, and the full denylist of families incompatible with VPC encryption-at-rest, see supported-instance-types.html. Do NOT replicate that list here — it changes quarterly.

Stable architectural notes (sizing-relevant):

  • OR1/OR2/OM2/OI2 migration is irreversible; min refresh interval 10s; bulk size 10 MB recommended.

  • Burstable (t3.*) is dev-only — CPU credits exhaust under sustained load.

Common Graviton search-instance specs (canonical RAM/vCPU; do NOT rederive — these are fixed):

InstancevCPURAM (GiB)EBS bandwidth
r7g.large.search216up to 5 Gbps
r7g.xlarge.search432up to 5 Gbps
r7g.2xlarge.search864up to 10 Gbps
r7g.4xlarge.search16128up to 12 Gbps
r7g.8xlarge.search3225612 Gbps
r7g.12xlarge.search4838420 Gbps
m7g.medium.search14up to 12.5 Gbps
m7g.large.search28up to 12.5 Gbps
m7g.xlarge.search416up to 12.5 Gbps

When deriving cluster topology, look up the RAM from this table — do NOT estimate it (r7g.2xlarge.search has 64 GiB RAM, not 16; r7g.4xlarge.search has 128 GiB, not 32). For instance families not listed (OR1, OR2, im4gn, etc.) verify against supported-instance-types.html.

UltraWarm tier
  • uw.medium cannot host k-NN graphs (lacks RAM headroom); use ultrawarm1.large for k-NN-on-warm.

  • Read-only; promote to hot for writes. Storage charge: primary shards only (no replica overhead). Recommended max shard size: 50 GiB. Requires dedicated cluster manager nodes.

  • For current SKUs and capacity per instance, see ultrawarm.html.

k-NN memory math

For FAISS HNSW float vectors with m=16:

bytes_per_vector ≈ 1.1 × (4 × dim + 8 × m)
total_memory ≈ bytes_per_vector × num_vectors × (1 + replicas)
Quick reference
VectorsDimMemory (replicas=1)Notes
1M384~3.5 GBSmall workload
1M768~6.7 GBBERT-class
10M768~67 GBMulti-node
100M768~670 GBMulti-node + maybe PQ
1M1536~13.4 GBOpenAI ada-002
10M1536~134 GBMulti-node

Native-index circuit breaker: default 50% of non-heap RAM. Verify against current knn-index/ doc for the exact percentage.

Engine impact:

  • Lucene engine: lighter, integrates fully with OpenSearch query DSL, best for filtered queries

  • FAISS HNSW: standard recall/latency trade-off, m=16 typical

  • FAISS HNSW + PQ: trade recall for ~4–32× memory savings

  • FAISS HNSW + scalar quantization (16-bit): 2× memory savings, minimal recall loss

  • FAISS IVF + PQ: best for batch-rebuild workloads (e.g., nightly index)

  • mode: "on_disk": graphs paged from disk; lower memory pressure, higher latency

k-NN UltraWarm constraints
  • NEVER use uw.medium for in-memory k-NN engines — instance lacks RAM headroom for k-NN graphs

  • Size so cumulative graph size of actively-searched shards ≤ knn.memory.circuit_breaker.limit × 61 GiB per uw.large

  • k-NN indexes can migrate to UltraWarm/cold from OS 2.17+

  • k-NN indexes do NOT force-merge to single segment during UltraWarm migration (keeps default 20 segments to avoid OOM)

OS 3.0 vector improvements

OS 3.0 introduces GPU-accelerated index build, derived-source vectors (reduced storage + faster cold start), concurrent segment search default-on for k-NN, and star-tree indexing for aggregations. For sizing impact, treat these as memory/storage reductions — verify under load with OpenSearch Benchmark; do not rely on vendor multiplier claims for capacity planning.

Serverless OCU sizing

OCU model
  • 1 OCU = 6 GiB RAM + matching vCPU + ~120 GiB ephemeral storage

  • Billing: per-second granularity, hourly rate

  • Indexing OCUs scale separately from search OCUs

Floors (NextGen and Classic)
ConfigurationIndexing floorSearch floorTotal billed
Redundancy ON (production default)1 OCU (0.5 × 2)1 OCU (0.5 × 2)4 × 0.5 OCU
Redundancy OFF (dev/test)0.5 OCU × 20.5 OCU × 22 × 0.5 OCU per workload type

Scale-to-zero: Serverless NextGen collections scale to zero OCUs when idle (an idle NextGen collection bills nothing for compute), whereas Classic collections hold the minimum OCU floor above at all times because Classic keeps warm capacity provisioned. Minimum-billing behavior can change per generation, so verify it against serverless-scaling.html (as the Caps section also references) before relying on it for cost projections. Where it holds, factor it into cost projections for intermittently-used collections: NextGen is materially cheaper when traffic is bursty or dev/test.

Caps

For current OCU defaults and account-level caps, see serverless-scaling.html.

Performance rules of thumb (skill IP — verify under load)
  • 1 indexing OCU ≈ 100–200 MB/s sustained ingest

  • 1 search OCU ≈ 50–200 simple QPS, 10–50 complex aggregations/sec

Critical Vector Search caveat

Vector Search collections CANNOT share OCUs with Search or TimeSeries collections — even with the same KMS key. Adding one vector collection roughly doubles the idle floor. Project both floors via https://calculator.aws.

If vector is exploratory, prefer running k-NN on existing Managed cluster instead of provisioning a separate Serverless Vector collection.

OpenSearch Ingestion (OSI) sizing

  • 1 OSI OCU = 6 GiB RAM + corresponding vCPU

  • Pricing: pay for OCUs allocated, regardless of data flow

  • Provisions Data Prepper 2.x (auto-upgraded within the 2.x line)

  • Persistent buffering steals OCUs from your declared max: 1:1 buffer-to-compute ratio. Raise max_units accordingly.

  • Common sources: OTel Collector, Fluent Bit, S3, Kinesis, MSK

  • All requests Sig v4 signed with osis:Ingest IAM permission

Cross-AZ data transfer

  • Within an AOS cluster: FREE (cluster manager / replica replication does NOT bill)

  • Between your VPC and AOS endpoint: billed at standard regional rates

  • NAT Gateway for plugins/Bedrock/external sources: $0.045/hr/AZ + $0.045/GB processed — use VPC endpoints for S3, Bedrock, STS to avoid

EBS storage (gp3 vs gp2)

  • gp3 is the default; ~9.6% cheaper than gp2

  • gp3 decouples IOPS from volume size; provisioned IOPS billed separately

  • AOS-managed gp3 list price differs from raw EBS gp3 — TCO calculators reusing raw EBS rate underestimate. Plug into https://calculator.aws.

Validate before cutover

Run OpenSearch Benchmark against the target cluster before cutover. The big5 workload is the standard search benchmark. The compare mode produces a baseline-vs-contender diff.

Manual snapshot S3 cost

  • Automated snapshots: stored in AOS-preconfigured S3 bucket, NO additional charge, kept 14 days

  • Manual / custom-retention / cross-region snapshots: stored in YOUR S3 bucket at standard S3 rates plus PUT charges

Sizing model addition: data_size × retention_days / 30 × $/GB-mo plus PUT cost.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an MCP server

Security

What our scanner noticed· Low risk

  • Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
AWS
Publisher
AWS (Official)
Source
aws/agent-toolkit-for-aws · plugins/aws-data-analytics/skills/amazon-opensearch-service · commit 0a218f8
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • NOTICE (added by AutoGineer from NOTICE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced
Also distributed in
aws/agent-toolkit-for-aws

AutoGineer is not affiliated with or endorsed by AWS.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download amazon-opensearch-service.zip