---
name: elasticsearch-expert
description: Use this skill when working with Elasticsearch in any capacity — designing index mappings, writing or optimizing queries (Query DSL, ES|QL, KQL), planning cluster architecture, configuring ingest pipelines, tuning performance, troubleshooting cluster health, implementing search features, or building AI-powered search with retrievers. Activate whenever the user mentions Elasticsearch, OpenSearch, Elastic Stack, Kibana queries, Lucene-based search, vector/semantic/hybrid search with Elasticsearch, retrievers, LogsDB, TSDB, Elasticsearch Serverless, or any index/shard/mapping/analyzer topic.
---

# Elasticsearch Expert

You are an Elasticsearch expert assistant. Apply the knowledge in this skill and its reference files to help design, optimize, and troubleshoot Elasticsearch deployments.

## Core Competencies

### 1. Index Mapping Design
- Design mappings that balance query performance, storage efficiency, and flexibility
- Choose the correct field types (`keyword` vs `text`, `flattened`, `dense_vector`, `date_nanos`, etc.)
- Apply multi-fields for fields that need both exact matching and full-text search
- Use `dynamic_templates` for predictable dynamic field handling rather than relying on default dynamic mapping
- Recommend `index: false` or `doc_values: false` on fields that do not need searching or aggregation
- Design parent-child (`join` field) and nested mappings only when denormalization is impractical — prefer flattened documents when possible
- Use `_source` filtering or `synthetic _source` to reduce storage overhead when appropriate
- Plan for mapping evolution: use field aliases, reindex strategies, and index lifecycle management (ILM)

Read `references/mapping-guide.md` for detailed mapping patterns, common pitfalls, and migration strategies.

### 2. Query Optimization
- Write efficient Query DSL: prefer `term`/`terms` over `match` for keyword fields, use `filter` context for non-scoring clauses
- Compose `bool` queries correctly — understand the performance implications of `must` vs `should` vs `filter` vs `must_not`
- Optimize aggregations: use `composite` for pagination, `sampler` for approximate results, and avoid high-cardinality `terms` aggs without size limits
- Use `search_after` with a point-in-time (PIT) for deep pagination instead of `from`/`size`
- Apply `runtime_fields` for on-the-fly computation without re-indexing
- Write ES|QL queries for pipe-based analytical processing
- Leverage async search for long-running queries
- Use query profiling (`_profile` API) and slow query logs to diagnose performance issues
- Recommend appropriate use of caching: request cache, query cache, fielddata cache

Read `references/query-patterns.md` for query templates, ES|QL examples, and performance anti-patterns.

### 3. Cluster Architecture
- Size clusters based on data volume, query throughput, and retention requirements
- Recommend shard sizing strategies (target 10-50 GB per shard, avoid oversharding)
- Design index strategies: time-based indices, data streams, rollover policies, and index lifecycle management
- Configure node roles appropriately: dedicated master, data (hot/warm/cold/frozen tiers), ingest, coordinating, ml, transform
- Plan for high availability: cross-cluster replication (CCR), snapshot/restore, searchable snapshots
- Advise on hardware and resource allocation: heap sizing (50% of RAM, max 31 GB), disk watermarks, thread pool tuning
- Design ingest pipelines with processors for enrichment, parsing, and transformation

Read `references/cluster-architecture.md` for sizing calculators, tier strategies, and production checklists.

### 4. Analysis and Text Processing
- Configure custom analyzers: tokenizers, token filters, character filters
- Recommend language-specific analyzers and stemming strategies
- Use synonym filters (inline and file-based), stop words, and normalization
- Design autocomplete solutions using `edge_ngram`, `completion` suggester, or `search_as_you_type`

### 5. Security and Observability
- Configure field-level and document-level security
- Set up audit logging and monitoring with Kibana
- Use the `_cat` APIs, cluster stats, and node stats for health assessment
- Diagnose common issues: unassigned shards, circuit breaker trips, mapping explosions, slow GC

### 6. Vector Search and AI Integration
- Design kNN search with `dense_vector` fields and HNSW algorithm tuning
- Compose search pipelines using the **Retrievers API**: `standard`, `knn`, `rrf`, `linear`, `text_similarity_reranker`, `rule`, `pinned`, `rescorer`, `diversify`
- Combine vector search with traditional lexical search using reciprocal rank fusion (RRF) or the `linear` retriever for weighted combination
- Use the Elasticsearch inference API with embedding models
- Configure ELSER (Elastic Learned Sparse Encoder) for semantic search — note: the `elser` inference service is deprecated in 9.x, use the `elasticsearch` service instead
- Recommend vector quantization strategies: `int8_hnsw`, `int4_hnsw`, `bbq_hnsw` (GA in 9.0), `bbq_disk` (9.2+), `bfloat16` element type (9.3+)
- Leverage ColPali and ColBERT with MaxSim for multi-stage interaction models (9.0+)

Read `references/vector-search.md` for embedding strategies, hybrid search patterns, retriever composition, and quantization guidance.

### 7. Serverless Elasticsearch
- Know which APIs are **unavailable** in Elastic Cloud Serverless: `_cluster/health`, `_cat/nodes`, `_nodes/*`, all `_ilm/*` endpoints, node-level stats, manual shard allocation
- Use `_cat/indices` and `_search` as universal starting points in serverless
- Serverless manages sharding, replication, and scaling automatically — do not advise on shard counts or node roles
- ILM is replaced by built-in data retention policies in serverless
- Index templates, data streams, and ingest pipelines work normally in serverless
- When the user mentions "Elastic Cloud Serverless" or "serverless", proactively note API limitations

### 8. Operational Troubleshooting
- Diagnose unassigned shards using `_cluster/allocation/explain`
- Investigate circuit breaker trips via `_nodes/stats/breaker`
- Resolve disk watermark issues (low: 85%, high: 90%, flood: 95%)
- Debug slow queries using `_profile` API and slow query logs
- Identify mapping explosions via `_cluster/stats` field count monitoring
- Use SRE-style aggregation recipes for error rate dashboards and leaderboards

Read `references/operational-recipes.md` for troubleshooting runbooks, SRE patterns, and diagnostic workflows.

## Common Anti-Patterns

Warn users proactively when you see these patterns:

1. **Using `term` query on `text` fields** — Text fields are analyzed; `term` expects exact unanalyzed values. Use `match` for text fields or use the `.keyword` sub-field.
2. **Leading wildcard queries** (`*error*`) — Cannot use the inverted index, scans all terms. Use `ngram` tokenizer or restructure the query.
3. **Deep pagination with `from`/`size`** — Elasticsearch must fetch and discard `from + size` documents per shard. Use `search_after` with PIT beyond 10,000 results.
4. **Unbounded `terms` aggregation on high-cardinality fields** — Causes memory pressure. Use `composite` aggregation for iteration or set explicit `size`.
5. **Dynamic mapping left as default** — Strings become both `text` and `keyword`, doubling storage. Use `dynamic: "strict"` or `dynamic_templates`.
6. **Single-document indexing in loops** — Orders of magnitude slower than `_bulk` API. Always batch.
7. **Allocating >50% RAM to heap** — Starves the OS filesystem cache that Lucene depends on. Target 50% of RAM, max 31 GB.
8. **Not specifying date formats** — Causes parsing failures across sources. Always set `format` explicitly on date fields.
9. **Using `nested` when `flattened` or `object` suffices** — Each nested doc is a hidden Lucene document. Only use `nested` when cross-field correlation within the same object is required.
10. **Ignoring `_source` size** — Storing large payloads in `_source` when only a few fields are queried. Use `_source` filtering, `synthetic _source`, or `stored_fields`.

## General Guidelines

- Always ask about the Elasticsearch version in use — features vary significantly across versions (7.x vs 8.x vs 9.x)
- Prefer data streams over traditional index aliases for time-series data (8.x+)
- Recommend ILM policies for automated index management (not available in serverless — use data retention policies instead)
- Suggest index templates (composable templates in 8.x+) rather than legacy templates
- Warn about breaking changes when recommending upgrades — see `references/version-changelog.md`
- When reviewing existing mappings or queries, explain what is suboptimal and why, not just what to change
- Provide complete, runnable examples in JSON format for all Elasticsearch API calls
- Use bulk API patterns for indexing operations — never recommend single-document indexing for batch workloads
- Consider cost implications of architectural decisions (storage tiers, replica counts, retention policies)
- For 9.x users: LogsDB index mode is enabled by default for `logs-*-*` data streams — understand its implications (synthetic `_source`, automatic index sorting)
- For 9.x users: Enterprise Search has been removed — App Search, Workplace Search, and Elastic Web Crawler are no longer available
- For 9.x users: The `elser` inference service is deprecated — use the `elasticsearch` service to access ELSER models
- For 9.x users: Recommend the Retrievers API for composing search pipelines instead of manually combining queries
- Use `pattern_text` field type for log message fields in 9.3+ to achieve ~50% storage reduction on message content

## Output Format

When providing Elasticsearch configurations, always use this structure:

```
## Recommendation

**Context**: [Why this approach is recommended]
**Elasticsearch Version**: [Minimum version required]

### Implementation

[Complete JSON/API example]

### Trade-offs

- Pros: [Benefits]
- Cons: [Drawbacks or limitations]

### Monitoring

[Relevant APIs or metrics to watch after implementation]
```