--- name: kibana-anomaly-detection description: > Elastic ML anomaly detection — investigation/RCA, score explanation, job lifecycle troubleshooting, and job operations. Use when answering "what broke?"/"which entity?"/RCA, "why is score high/low?"/renormalization, "datafeed stopped"/"memory limit"/hard_limit, or configuring ML anomaly detection jobs. Reads results from `.ml-anomalies-*` and job state from ML REST APIs. metadata: author: elastic version: 0.3.0 universal: true compatibility: Elasticsearch 8.x–9.x or Elastic Cloud Serverless with ML anomaly detection; Kibana 8.x–9.x for saved-object context only --- # Elastic ML Anomaly Detection Expert process for ML anomaly detection: attribute incidents to entities, explain scores and model behavior, diagnose job lifecycle failures, and manage jobs. Read anomaly **results** from `POST /.ml-anomalies-*/_search` (Serverless-safe) and **job/datafeed state** from ML REST APIs. When the user embeds fixture evidence (influencer rows, job stats) in the prompt, apply the judgment below directly — do not re-fetch fields already supplied. ## Environment Configuration This skill executes Elasticsearch operations through the `elastic` CLI. If the [`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds. This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`, `GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API directly. ## Mode selector | User intent | Mode | | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | "What broke?" / RCA / cross-job / blast radius / influencers / log categories | **Investigate** | | "Why score high/low?" / renormalization / model bounds / forecasts | **Explain** | | Missing docs / memory limit / datafeed stopped / lifecycle / calendars | **Troubleshoot** | | Create a job / configure a datafeed / start analysis / retrieve results | **Manage** | | Security framing (attack chains, MITRE, exfil) | Investigate + [references/security-anomaly-expert.md](references/security-anomaly-expert.md) | | Observability/SRE framing (degradation, capacity, deployment regression) | Investigate + [references/observability-anomaly-expert.md](references/observability-anomaly-expert.md) | When a question spans modes: **Investigate → Explain → Troubleshoot**. Finish one mode before blending logic. > **Serverless note:** Legacy `/_ml/anomaly_detectors/{job_id}/results/*` endpoints return HTTP 410 in Serverless. > Always query `.ml-anomalies-*` via `POST /.ml-anomalies-*/_search` with `result_type` filters. ## Score quick reference - `record_score` bands: **>75** critical · **50–75** warning · **25–50** minor · **<25** informational - `multi_bucket_impact ≥ 3` → sustained shift (not a transient spike) - `initial_record_score >> record_score` → renormalization (model saw worse anomalies later) - `actual << typical` with `count`/`low_count`/`low_mean` → absence/outage, not just a low value - Low scores across many jobs > one high score — composite cross-job signal often beats single-detector severity > Full score definitions, renormalization mechanics, and `anomaly_score_explanation` components: > [references/score-reference.md](references/score-reference.md). ## Core concepts Treat `.ml-anomalies-*` as layered result types via `result_type` in search queries: | `result_type` | Scope | Key fields | | --------------------- | --------------- | ---------------------------------------------------------------------------------------- | | `bucket` | Time window | `anomaly_score`, `initial_anomaly_score`, `timestamp` | | `record` | Detector row | `record_score`, `initial_record_score`, `actual`, `typical`, `anomaly_score_explanation` | | `influencer` | Entity × bucket | `influencer_field_name`, `influencer_field_value`, **`influencer_score`** | | `model_plot` | Bounds | `model_lower`, `model_upper`, `actual` | | `category_definition` | Log patterns | `category_id`, `terms`, `regex`, `examples` | Read scores this way: - `anomaly_score` / `record_score` = **current normalized** values (move as the model sees new extremes). - `initial_anomaly_score` / `initial_record_score` = **immutable snapshots** from detection time. - **`influencer_score` ranks entity responsibility within a bucket** — the highest score is the primary suspect, not the bucket-level `anomaly_score` alone. - Map entities via `partition_field_value` / `by_field_value` / `over_field_value`. - Read `multi_bucket_impact` (-5 to +5) to separate single-bucket spikes from sustained trends. --- ## Mode: Investigate — RCA **When:** "what broke?", "which entity caused this?", cross-job correlation, blast radius, attack/cascade chains. ### Process 1. **Discover jobs.** Call `GET /_ml/anomaly_detectors` when the job ID is unknown. Call `GET /_ml/anomaly_detectors/{job_id}` and `GET /_ml/datafeeds/datafeed-{job_id}` to learn source indices, entity fields (`by_field_name`, `over_field_name`, `partition_field_name`), and `bucket_span`. The decision: identify the related job group — jobs sharing a datafeed index or entity field monitor the same system from different angles. 2. **Scope the incident window.** Call `POST /.ml-anomalies-*/_search` with `result_type: bucket`, a time range, and optional minimum `anomaly_score`. The decision: fix the incident start/end and count how many jobs co-fire in that window. Low scores across many jobs simultaneously often indicate a systemic root cause. 3. **Attribute to entities (critical for RCA).** For the anomalous bucket timestamp, call `POST /.ml-anomalies-*/_search` with `result_type: influencer`, the job ID(s), and the bucket time range. Sort by **`influencer_score` descending**. The decision: name the entity with the **highest `influencer_score`** as the likely cause — it ranks how unusual each entity is in that bucket. Do not restate only the bucket `anomaly_score` without attributing responsibility. Recommend drilling into that entity's records next. 4. **Cross-job confirmation.** Re-query influencers (or bucket records) across related job IDs for the same entity values and time window. Entities anomalous in **2+ jobs** are prime suspects (resource fault or systemic failure); single-job entities are often downstream victims. See [references/protocols/investigation.md](references/protocols/investigation.md). 5. **Drill into records.** Call `POST /.ml-anomalies-*/_search` with `result_type: record`, exact job ID, entity filters (`partition_field_value`, `by_field_value`), and low minimum `record_score` (25 or lower). Read `multi_bucket_impact ≥ 3` as sustained behavioral shift. Read `actual` vs `typical` for fault class (spike vs absence/outage). 6. **Confirm with source evidence.** Call `POST /{index}/_search` on the datafeed source index for the suspect entity and time window. Raw source documents are ground truth — never close an RCA without them. 7. **Synthesize.** Report: **root cause entity · affected jobs · temporal progression · fault class · severity · recommended actions**. Worked walkthrough: [references/worked-example.md](references/worked-example.md). Query templates: [references/investigation-queries.md](references/investigation-queries.md). ### Rules 1. **Rank by `influencer_score`, not `anomaly_score`, for "which entity?"** — bucket score is aggregate; influencer score attributes cause. 2. **Multi-job entities are prime suspects; single-job entities are usually victims.** 3. **Earliest anomaly timestamp wins** — reconstruct chronology from record timestamps across jobs. 4. **`multi_bucket_impact ≥ 3` = sustained behavioral shift**, weight higher than transient spikes. 5. **Use low score thresholds (25 or lower) for influencer/record queries** — high thresholds miss correlated entities. 6. **Never close an RCA without source evidence** from the datafeed index. --- ## Mode: Explain — Score / model behavior **When:** "why is my score 30/90?", "score dropped overnight", "what is renormalization?", "why wasn't this detected?". ### Process 1. **Decide fetch vs interpret.** If the user supplies a record with `record_score`, `initial_record_score`, `actual`, and `typical`, interpret directly. Otherwise load config with `GET /_ml/anomaly_detectors/{job_id}` and records with `POST /.ml-anomalies-*/_search` (`result_type: record`). 2. **Always show both `initial_record_score` and `record_score`.** The gap is the renormalization story. Large positive drift (`initial_record_score >> record_score`) means a later, more extreme anomaly rescale this record downward — expected healthy behavior, not a broken model. 3. **Classify the pattern before speculating.** | Pattern | Interpretation | | ------------------------------------------------------------ | ----------------------------------------------------------------- | | `initial_record_score >> record_score` | Renormalization — explain before suggesting config changes | | `actual << typical` with `low_count`/`count`/`low_mean` | Absence/outage anomaly — investigate the outage, not score tuning | | `high_variance_penalty: true` in `anomaly_score_explanation` | Noisy metric — wide bounds absorbed the spike | | `incomplete_bucket_penalty: true` | Ingest lag or sparse bucket — score legitimately reduced | Only cite `anomaly_score_explanation` factors **present** in the record. 4. **Quantify renormalization (optional).** Re-query records sorted by `timestamp`; compute `score_drift = initial_record_score - record_score` and flag large drift. 5. **Add visual context when needed.** If `model_plot_config.enabled`, query `result_type: model_plot` and compare `actual` to `model_lower`/`model_upper`. For categorization jobs, query `result_type: category_definition`. 6. **Check job health when scores look wrong persistently.** Call `GET /_ml/anomaly_detectors/{job_id}/_stats` — `model_size_stats.memory_status` of `hard_limit` corrupts learning and can invalidate scores. Escalate to Troubleshoot mode. ### `anomaly_score_explanation` components | Component | Effect | What it means | | -------------------------------- | ------- | ------------------------------------------------------------ | | `anomaly_length` | ↑ score | More consecutive anomalous buckets | | `single_bucket_impact` | ↑ score | Lower probability → higher impact | | `multi_bucket_impact` | ↑ score | Sustained pattern contribution | | `anomaly_characteristics_impact` | ↑ score | Mean shift vs. variance change | | `high_variance_penalty` | ↓ score | Noisy data → wide bounds → anomaly less surprising | | `incomplete_bucket_penalty` | ↓ score | Bucket has less data than expected (ingest lag, sparse data) | ### Rules 1. **Explain renormalization before diagnosing config** — score drift is the most common "score dropped" cause. 2. **`actual << typical` with count/low_count is an absence anomaly** — distinguish outages from value spikes. 3. **Weekly seasonality needs ≥3 weeks of training data** — flag young jobs as the cause. 4. **Detector function direction matters** — see [references/anomaly-detection-functions.md](references/anomaly-detection-functions.md). --- ## Mode: Troubleshoot — Job lifecycle **When:** "missing documents", "datafeed stopped", **`hard_limit`**, "results look wrong", lifecycle changes. ### Process 1. **Load job and datafeed state.** Call `GET /_ml/anomaly_detectors/{job_id}/_stats` and `GET /_ml/datafeeds/datafeed-{job_id}/_stats`. Read `state`, `data_counts`, **`model_size_stats`**, and datafeed `state`. If the user embeds stats JSON, diagnose from `memory_status` and datafeed state directly. 2. **Diagnose memory status first (critical).** Inspect `model_size_stats`: | Field | Meaning | | -------------------------- | ----------------------------------------------------------- | | `memory_status` | `ok` / `soft_limit` (pruning) / **`hard_limit` (critical)** | | `model_bytes` | Current memory used | | `model_bytes_memory_limit` | Configured `model_memory_limit` | When **`memory_status` is `hard_limit`** and `model_bytes` equals `model_bytes_memory_limit`, the model hit its memory ceiling — it stops learning new entities and results degrade or stop. A stopped datafeed is often a **symptom**, not the root cause. **Do not recommend only restarting the datafeed** — that alone does not clear a hard limit. 3. **Remediate hard_limit.** The fix is to **raise `model_memory_limit`** (via job update) **and/or reduce model size** by lowering cardinality (fewer partition/by/over field values, split into multiple jobs). Raising the limit requires the lifecycle sequence below (stop datafeed → close job → update → open → start). Optionally call `POST /_ml/anomaly_detectors/_estimate_model_memory` to size the new limit from source cardinality. 4. **Diagnose missing documents / query timing.** After memory is healthy, inspect datafeed `query_delay` and `delayed_data_check_config` via `GET /_ml/datafeeds/datafeed-{job_id}`. Search `.ml-annotations-*` for delayed-data events. Set `query_delay` to P95 ingest latency + buffer (default `60s`–`120s`). 5. **Read job messages.** Search `.ml-notifications-*` for the job ID when errors are unclear. 6. **Recover corrupted model state.** Call `POST /_ml/anomaly_detectors/{job_id}/model_snapshots/{snapshot_id}/_revert` to revert to a known-good snapshot when the model was corrupted during hard_limit. ### Lifecycle for config changes (memory limit, query_delay) Apply in order — skipping steps causes rejected updates: 1. `POST /_ml/datafeeds/datafeed-{job_id}/_stop` 2. `POST /_ml/anomaly_detectors/{job_id}/_close` 3. `POST /_ml/anomaly_detectors/{job_id}/_update` (memory limit) and/or `POST /_ml/datafeeds/datafeed-{job_id}/_update` (query_delay) 4. `POST /_ml/anomaly_detectors/{job_id}/_open` 5. `POST /_ml/datafeeds/datafeed-{job_id}/_start` Preview changes with `POST /_ml/datafeeds/datafeed-{job_id}/_preview` before restarting. > **`hard_limit` corrupts model state** and causes downstream missing-doc false alarms. **Fix memory before fixing > `query_delay`.** Full troubleshooting detail: > [references/troubleshooting-reference.md](references/troubleshooting-reference.md). ### Rules 1. **Ground lifecycle diagnosis in `memory_status`** — not generic "restart it" advice. 2. **Fix memory before `query_delay`** — hard_limit invalidates downstream diagnostics. 3. **Stop datafeed → close job → update → open → start** for any memory or datafeed config change. 4. **Do not delete the job** as first remediation for hard_limit — raise limit and/or reduce cardinality. --- ## Mode: Manage — Create / configure jobs **When:** "set up a job", "create an ML detector", "monitor X over time". For the full create/open/start lifecycle, prefer the `elasticsearch-anomaly-detection` skill. This mode summarizes the sequence and detector selection: 1. **Verify target index.** Call `GET /{index}/_mapping` — confirm time field and detector fields exist. 2. **Create job.** Call `PUT /_ml/anomaly_detectors/{job_id}` with `analysis_config` (detectors, `bucket_span`, influencers) and `data_description.time_field`. 3. **Create datafeed.** Call `PUT /_ml/datafeeds/datafeed-{job_id}` with `indices`, `query`, and `query_delay`. 4. **Open and start.** Call `POST /_ml/anomaly_detectors/{job_id}/_open`, then `POST /_ml/datafeeds/datafeed-{job_id}/_start`. 5. **Confirm.** Call `GET /_ml/anomaly_detectors/{job_id}/_stats` and `GET /_ml/datafeeds/datafeed-{job_id}/_stats`. Choose detector functions from user intent — see [references/anomaly-detection-functions.md](references/anomaly-detection-functions.md). Worked JSON bodies: [references/job-creation-recipes.md](references/job-creation-recipes.md). ### Rules 1. **Create job before datafeed.** Open job before starting datafeed. 2. **`query_delay` = P95 ingest latency + buffer** (60s–120s safe default). 3. **`by_field_name` vs `over_field_name`:** `by` compares entity to its own history; `over` compares to peer group. 4. **Forecasts require non-population jobs** — jobs with `over_field_name` cannot be forecasted. --- ## Examples **RCA:** "Something caused a spike in checkout latency — which entity?" → Query influencers for the bucket → **web-07** has highest `influencer_score` (91.5) vs 22.0 and 8.4 → name web-07 as likely cause → recommend drilling into its records — do not answer with only bucket `anomaly_score` 88. **Score drop:** "Score went from 90 to 55 — did the model change?" → Compare `initial_record_score` vs `record_score` → explain renormalization if drift is large. **Memory limit:** "Job shows `hard_limit` and datafeed stopped." → Diagnose `model_size_stats.memory_status = hard_limit` → raise `model_memory_limit` via close/update/open lifecycle and/or reduce cardinality — **not** "just restart the datafeed". **New job:** "Detect unusual error rates per host." → `high_count` with `by_field_name: host.keyword` → create/open/start sequence. --- ## Guidelines 1. **Pick a mode first.** Don't blend RCA logic with score-explanation logic in one response. 2. **For "which entity?" rank `influencer_score`**, not bucket `anomaly_score`. 3. **For lifecycle failures read `memory_status`** before recommending datafeed restarts. 4. **Show `initial_record_score` alongside `record_score`** — the gap tells the renormalization story. 5. **Fix memory before `query_delay`.** Hard_limit invalidates downstream diagnostics. 6. **Confirm RCAs with source evidence** from the datafeed index. ## Operations | HTTP API (shorthand) | `elastic` CLI command | | ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | | `GET /{index}/_mapping` | `elastic es indices get-mapping --index ''` | | `POST /{index}/_search` | `elastic es search --index '' --input-file ''` | | `GET /_ml/anomaly_detectors` | `elastic es ml get-jobs` | | `GET /_ml/anomaly_detectors/{job_id}` | `elastic es ml get-jobs --job-id ''` | | `GET /_ml/anomaly_detectors/{job_id}/_stats` | `elastic es ml get-job-stats --job-id ''` | | `GET /_ml/datafeeds/datafeed-{job_id}` | `elastic es ml get-datafeeds --datafeed-id 'datafeed-'` | | `GET /_ml/datafeeds/datafeed-{job_id}/_stats` | `elastic es ml get-datafeed-stats --datafeed-id 'datafeed-'` | | `POST /.ml-anomalies-*/_search` | `elastic es search --index '.ml-anomalies-*' --input-file ''` | | `POST /.ml-annotations-*/_search` | `elastic es search --index '.ml-annotations-*' --input-file ''` | | `POST /.ml-notifications-*/_search` | `elastic es search --index '.ml-notifications-*' --input-file ''` | | `POST /_ml/anomaly_detectors/_estimate_model_memory` | `elastic es ml estimate-model-memory --analysis-config ''` | | `PUT /_ml/anomaly_detectors/{job_id}` | `elastic es ml put-job --job-id '' --input-file ''` | | `PUT /_ml/datafeeds/datafeed-{job_id}` | `elastic es ml put-datafeed --datafeed-id 'datafeed-' --input-file ''` | | `POST /_ml/anomaly_detectors/{job_id}/_open` | `elastic es ml open-job --job-id ''` | | `POST /_ml/anomaly_detectors/{job_id}/_close` | `elastic es ml close-job --job-id ''` | | `POST /_ml/anomaly_detectors/{job_id}/_update` | `elastic es ml update-job --job-id '' --analysis-limits ''` | | `POST /_ml/datafeeds/datafeed-{job_id}/_update` | `elastic es ml update-datafeed --datafeed-id 'datafeed-' --input-file ''` | | `POST /_ml/datafeeds/datafeed-{job_id}/_start` | `elastic es ml start-datafeed --datafeed-id 'datafeed-'` | | `POST /_ml/datafeeds/datafeed-{job_id}/_stop` | `elastic es ml stop-datafeed --datafeed-id 'datafeed-'` | | `POST /_ml/datafeeds/datafeed-{job_id}/_preview` | `elastic es ml preview-datafeed --datafeed-id 'datafeed-'` | | `POST /_ml/anomaly_detectors/{job_id}/model_snapshots/{snapshot_id}/_revert` | `elastic es ml revert-model-snapshot --job-id '' --snapshot-id ''` | Search body shapes for each `result_type` and troubleshooting queries are documented in [references/investigation-queries.md](references/investigation-queries.md) and [references/troubleshooting-reference.md](references/troubleshooting-reference.md).