--- name: kibana-anomaly-detection description: Elastic ML anomaly detection skill — investigation/RCA, score explanation, job operations (create, datafeed, start/stop, results), and troubleshooting (missing docs, memory limits, datafeed health, lifecycle). Operates against Kibana Agent Builder MCP tools (`ad_*`) on `.ml-anomalies-*`, `.ml-config`, `.ml-notifications-*`, `.ml-annotations-*`. Use when answering "what broke?"/"which entity?"/RCA, "why is score high/low?"/renormalization, "datafeed stopped"/"memory limit", or any request to set up or configure an ML anomaly detection job. metadata: author: elastic version: 0.2.0 compatibility: Kibana 8.x–9.x with Agent Builder and Workflows; Elasticsearch 8.x–9.x with machine learning --- # Elastic ML Anomaly Detection Single skill covering all anomaly detection work against **Kibana Agent Builder** MCP at `{KIBANA_URL}/api/agent_builder/mcp`. Use the **Mode Selector** below to pick the right approach for the user's question — modes share the same tool surface and concepts. ## Platform - Read path: ES|QL against `.ml-anomalies-*`, `.ml-config`, `.ml-notifications-*`, `.ml-annotations-*` - Always-available: `platform.core.execute_esql` (plus additional platform tools for search, index mapping, and documentation — see `scripts/agent_builder_constants.json`) - ML API spec (if available): `.kibana_ai_openapi_spec_elasticsearch` — see [references/anomaly-detection-openapi-spec-discover.md](references/anomaly-detection-openapi-spec-discover.md) for discovery pattern. - **Run `ad_validate_ml_tool_permissions` first** when tools return empty/misleading results — missing privileges are the most common cause of false negatives. Full permissions matrix: [references/permissions-matrix.md](references/permissions-matrix.md). ## Mode Selector | User intent | Mode | | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | "What broke?" / RCA / cross-job / blast radius / influencers / log categories | **Investigate** | | "Why score high/low?" / renormalization / model bounds / forecasts | **Explain** | | Missing docs / memory limit / datafeed stopped / CCS / lifecycle / calendars | **Troubleshoot** | | Create a job / configure a datafeed / start analysis / retrieve results | **Manage** | | Security framing (attack chains, MITRE, exfil) | Investigate + [references/security-anomaly-expert.md](references/security-anomaly-expert.md) | | Observability/SRE framing (degradation, capacity, deployment regression) | Investigate + [references/observability-anomaly-expert.md](references/observability-anomaly-expert.md) | When a question spans modes: **Investigate → Explain → Troubleshoot**. Don't blend mode logic — finish one before moving on. --- ## Score Quick Reference - `record_score` bands: **>75** critical · **50–75** warning · **25–50** minor · **<25** informational - `multi_bucket_impact ≥ 3` → sustained shift (not a transient spike) - `initial_record_score >> record_score` → renormalization (model saw worse anomalies later) - `actual << typical` with `count`/`low_count`/`low_mean` → absence/outage, not just low value - Low scores across many jobs > one high score — composite cross-job signal often beats single-detector severity > Full score definitions, renormalization mechanics, and `anomaly_score_explanation` components: > [references/score-reference.md](references/score-reference.md). ## Core concepts Treat `.ml-anomalies-*` as three layers, accessed via `result_type`: - **`bucket`** — bucket-level unusualness per `bucket_span`. `anomaly_score` is the aggregate across all detectors. - **`record`** — finest-grained rows with `actual` vs `typical`, `probability`, `record_score`, `anomaly_score_explanation`. - **`influencer`** — entity contributions ranked within a bucket (`influencer_score`). Read scores this way: - `anomaly_score` / `record_score` = **current normalized** values (move as the model sees new extremes). - `initial_anomaly_score` / `initial_record_score` = **immutable snapshots** from detection time. - Compare `actual` to `typical`; use `probability` for raw likelihood. - Map entities via `partition_field_value` / `by_field_value` / `over_field_value`. - Read `multi_bucket_impact` (-5 to +5) to separate single-bucket spikes from sustained trends. --- ## Mode: Investigate — RCA **When:** "what broke?", "which entity caused this?", cross-job correlation, blast radius, attack/cascade chains. ### Tool chain | Phase | Tools | | --------------------- | -------------------------------------------------------------------------------------------------------------- | | Discovery | `ad_get_available_metadata`, `ad_get_jobs`, `ad_discover_related_jobs`, `ad_discover_jobs_by_datafeed_index` | | Timeline / scope | `ad_query_anomaly_timeline` | | Cross-job / entities | `ad_rca_cross_job_entity_match`, `ad_rca_multi_job_entities`, `ad_rca_entity_profile` | | Records / influencers | `ad_query_anomaly_records`, `ad_query_influencers` | | RCA depth | `ad_rca_detector_fingerprint`, `ad_rca_correlation`, `ad_rca_blast_radius`, `ad_rca_score_reassessment` | | Evidence / categories | `ad_get_job_datafeed_config`, `ad_rca_source_evidence`, `ad_get_categories`, `ad_search_log_category_examples` | ### Protocol Follow the 14-step sequence in [references/protocols/investigation.md](references/protocols/investigation.md). High level: `ad_get_available_metadata` → pair `ad_discover_jobs_by_datafeed_index` with `ad_discover_related_jobs` → `ad_query_anomaly_timeline` → rank with `ad_rca_multi_job_entities` (`min_job_count=2`) → `ad_rca_detector_fingerprint` → drill with `ad_query_anomaly_records` + `ad_query_influencers` (low `min_score=25`) → profile with `ad_rca_entity_profile` → order with `ad_rca_correlation` → confirm with `ad_rca_source_evidence`. When `by_field_name == "mlcategory"`, compare with `ad_get_categories` + paired `ad_search_log_category_examples` (baseline vs. anomaly window). Finish with a written RCA: **root cause entity · affected jobs · temporal progression · fault class (resource/network/application) · severity · recommended actions**. Worked example: [references/worked-example.md](references/worked-example.md). Full ES|QL templates and parameters: [references/investigate-anomaly-esql-tools.md](references/investigate-anomaly-esql-tools.md). ### Rules 1. **Multi-job entities are prime suspects; single-job entities are usually victims.** Use `min_job_count=2`. 2. **Earliest anomaly timestamp wins** — sort `ad_rca_correlation` by timestamp; first-appearing entity = origin. 3. **`multi_bucket_impact ≥ 3` = sustained behavioral shift**, weight higher than transient spikes. 4. **Never close an RCA without `ad_rca_source_evidence`** — raw source documents are ground truth. 5. **Use low `min_score` (25 or lower) for influencer queries** — high thresholds miss correlated entities. --- ## Mode: Explain — Score / model behavior **When:** "why is my score 30/90?", "score dropped overnight", "what is renormalization?", "why wasn't this detected?". ### Score types | Field | Scope | Meaning | | ---------------------- | --------------- | ----------------------------------------------------------------------- | | `record_score` | Single record | Normalized severity after renormalization. | | `initial_record_score` | Single record | Score at detection time. Gap vs `record_score` = renormalization drift. | | `anomaly_score` | Bucket | Aggregate severity across all detectors in a bucket. | | `influencer_score` | Entity × bucket | How anomalous a specific entity is in that bucket. | ### `anomaly_score_explanation` components | Component | Effect | What it means | | -------------------------------- | ------- | ------------------------------------------------------------ | | `anomaly_length` | ↑ score | More consecutive anomalous buckets | | `single_bucket_impact` | ↑ score | Lower probability → higher impact | | `multi_bucket_impact` | ↑ score | Sustained pattern contribution | | `anomaly_characteristics_impact` | ↑ score | Mean shift vs. variance change | | `high_variance_penalty` | ↓ score | Noisy data → wide bounds → anomaly less surprising | | `incomplete_bucket_penalty` | ↓ score | Bucket has less data than expected (ingest lag, sparse data) | ### Why a score looks wrong - **Unexpectedly low:** `high_variance_penalty`, renormalization, <3 weeks training for weekly seasonality, `bucket_span` too large, wrong detector function (`mean` vs `high_mean`), `incomplete_bucket_penalty`, suppression by `custom_rules`. - **Unexpectedly high:** insufficient history (early training over-flags), high-cardinality split (too few points per entity), `use_null: true` on a sparse field. ### Tool chain | Purpose | Tools | | ---------------------- | ---------------------------------------------------------------------------------- | | Records + explanation | `ad_query_anomaly_records` (exact `job_id_pattern`) | | Renormalization drift | `ad_rca_score_reassessment` (`score_drift = initial_record_score - record_score`) | | Model bounds (visual) | `ad_get_model_plot` — actual outside `model_lower`/`model_upper` = anomaly | | Forecast overlap | `ad_get_forecast_results` | | Influencer attribution | `ad_query_influencers` | | Config & detector | `ad_get_job_datafeed_config` — `bucket_span`, function, `custom_rules`, `use_null` | | Categorization | `ad_get_categories` | | Model snapshots | `ad_get_model_snapshots` | | Structured diagnostic | **`ad_wf_troubleshoot_anomaly_score`** (full decision tree) | ### Decision tree (`ad_wf_troubleshoot_anomaly_score`) 1. `ad_get_jobs` — ≥3 weeks data for weekly seasonality? 2. `ad_ts_model_memory_health` — `memory_status` healthy? 3. `ad_ts_delayed_data_annotations` — no incomplete buckets? 4. `ad_query_anomaly_records` — compare `record_score` vs `initial_record_score`. 5. `ad_get_job_datafeed_config` — `bucket_span`, detector function, `custom_rules`, `use_null`. 6. `ad_get_model_plot` — wide bounds → `high_variance_penalty`. 7. `ad_rca_score_reassessment` — renormalization drift across history. 8. Explain `anomaly_score_explanation` factors. ### Rules 1. **Always show both `initial_record_score` and `record_score`** — the gap is the renormalization story. 2. **Explain renormalization before diagnosing config** — score drift is the most common "score dropped" cause and needs no config change. 3. **`actual << typical` with `count`/`low_count` is an absence anomaly** — distinguish outages from value spikes. 4. **`high_variance_penalty` and `incomplete_bucket_penalty` explain most "low score" surprises** without remediation. 5. **Weekly seasonality needs ≥3 weeks of training data** — flag young jobs as the cause. For detector function selection details, see [references/anomaly-detection-functions.md](references/anomaly-detection-functions.md). --- ## Mode: Troubleshoot — Job ops **When:** "missing documents", "datafeed stopped", "hard_limit", "results look wrong", lifecycle changes, calendars, CCS. ### Common issues → fast paths | Issue | Fast path | Full decision tree | | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | | Missing docs / `query_delay` warning | `ad_ts_delayed_data_annotations` → `ad_ts_bucket_event_gaps` → `ad_ts_ingest_latency_estimate` → `ad_update_datafeed_query_delay` | `ad_wf_troubleshoot_query_delay` | | Memory `soft_limit` / `hard_limit` | `ad_ts_model_memory_health` → `ad_wf_ts_field_cardinality` → `ad_estimate_memory_requirement` → `ad_update_model_memory_limit` | `ad_wf_troubleshoot_memory_limit` | | Datafeed not running / job state | `ad_get_jobs` (state) → `ad_get_job_messages` → `ad_manage_datafeed` | — | | CCS / `remote_cluster:` indices | `ad_ts_ccs_diagnostics` | — | | Score sanity check | — | `ad_wf_troubleshoot_anomaly_score` | > `hard_limit` corrupts model state and causes downstream missing-doc false alarms (categorizer silently skips events > for unknown categories). **Fix memory before fixing `query_delay`.** ### Memory concepts | Field | Meaning | | ----------------------------------- | ------------------------------------------------------- | | `model_bytes` | Current memory used | | `peak_model_bytes` | High-water mark since job opened | | `model_bytes_memory_limit` | Configured `model_memory_limit` | | `memory_status` | `ok` / `soft_limit` (pruning) / `hard_limit` (critical) | | `total_by_field_count > 100k` | `by_field` cardinality too high — dominant driver | | `total_partition_field_count > 10k` | Partition explosion | | `total_category_count > 10k` | Too many distinct log patterns | Prefer **`ad_estimate_memory_requirement`** (samples cardinality from source, calls Estimate Model Memory API) over heuristics like `peak_model_bytes * 1.3` — the heuristic ignores pure influencer and categorization memory. ### Datafeed & timing concepts - **`query_delay`** — how far behind real time the datafeed queries. Too small → missing docs; too large → slower alerts. Set to **P95 ingest latency + buffer** (default `60s`–`120s`). - **`delayed_data_check_config`** — how aggressively the datafeed checks for late data. - **`bucket_span`** — analysis interval. Align with data granularity and detection window. - **`frequency`** — defaults to `min(query_delay, bucket_span / 2)`. ### Lifecycle for config changes (memory limit, query_delay) 1. Stop datafeed: `ad_manage_datafeed` (`action=_stop`) 2. Close job 3. Update config: `ad_update_model_memory_limit`, `ad_update_datafeed_query_delay`, `ad_update_delayed_data_check_config` 4. Open job: `ad_open_job` 5. Start datafeed: `ad_manage_datafeed` (`action=_start`) Recover a corrupted period without resetting the whole model: `ad_revert_model_snapshot`. ### Tool surface | Category | Tools | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Permissions / metadata | `ad_validate_ml_tool_permissions`, `ad_get_available_metadata`, `ad_get_jobs` | | Job + datafeed state | `ad_get_job_datafeed_config`, `ad_get_job_messages`, `ad_manage_datafeed`, `ad_preview_datafeed_with_latency` | | Timing / missing docs | `ad_ts_delayed_data_annotations`, `ad_ts_bucket_event_gaps`, `ad_ts_ingest_latency_estimate`, `ad_update_datafeed_query_delay`, `ad_update_delayed_data_check_config`, `ad_wf_troubleshoot_query_delay` | | Memory | `ad_ts_model_memory_health`, `ad_wf_ts_field_cardinality`, `ad_estimate_memory_requirement`, `ad_update_model_memory_limit`, `ad_wf_troubleshoot_memory_limit` | | Model / lifecycle | `ad_get_model_snapshots`, `ad_revert_model_snapshot`, `ad_open_job`, `ad_create_job` | | CCS | `ad_ts_ccs_diagnostics` | | Calendars | `ad_get_calendar_events`, `ad_create_calendar_event` | Full parameter tables, ES|QL templates, and REST step lists: [references/troubleshoot-anomaly-tool-reference.md](references/troubleshoot-anomaly-tool-reference.md). ### Rules 1. **`ad_validate_ml_tool_permissions` first** — missing privileges produce misleading empty results. 2. **Fix memory before `query_delay`** — `hard_limit` corrupts state; `query_delay` fixes on a memory-limited job are wasted. 3. **Stop the datafeed before updating it.** Updating a running datafeed is rejected. 4. **Close the job before updating memory limit.** Sequence above. 5. **Prefer workflow tools (`ad_wf_*`) over manually chaining diagnostics** for complex decisions. 6. **`ad_preview_datafeed_with_latency` before starting** — confirm the datafeed returns data after config changes. --- ## Mode: Manage — Create / configure jobs **When:** "set up a job", "create an ML detector", "monitor X over time", "detect rare/unusual/anomalous values". ### 4-step workflow ```text PUT _ml/anomaly_detectors/ # 1. Define job (ad_create_job) PUT _ml/datafeeds/datafeed- # 2. Define datafeed (ad_create_datafeed) POST _ml/anomaly_detectors//_open # 3a. Open job (ad_open_job) POST _ml/datafeeds/datafeed-/_start # 3b. Start datafeed (ad_manage_datafeed action=_start) GET _ml/anomaly_detectors//results/records # 4. Read results ``` ### Process 1. **Build configs.** Parse the user request into job + datafeed JSON with no null fields. 2. **Apply smart defaults:** | Field | Default | Override when | | ---------------- | --------------------------------------- | ------------------------------------------------- | | `bucket_span` | `"15m"` | User specifies a different span | | `time_field` | `"@timestamp"` | User names a different timestamp field | | `index` | `"logs-*"` | User specifies an index or pattern | | `datafeed_query` | `{"match_all": {}}` | User mentions filters, processes, or time windows | | `influencers` | by/over/partition fields from detectors | User adds extra influencer fields | | `job_id` | Generated from user description | User provides an explicit ID | | `query_delay` | `"60s"` | P95 ingest latency is higher | 3. **Choose detector function** from user intent — full table in [references/anomaly-detection-functions.md](references/anomaly-detection-functions.md): - "high CPU" / "unusually large" → `high_mean` or `high_sum` - "rare logins" / "unusual values" → `rare` (variants below) - "too many requests" / "spike in count" → `high_count` `rare` variants: - Infrequent globally → `rare by_field_name: X` - Infrequent vs peers → `rare by_field_name: X over_field_name: Y` - Infrequent per segment → `rare by_field_name: X partition_field_name: Y` - Infrequent per segment vs peers → `rare by_field_name: X over_field_name: Y partition_field_name: Z` 4. **Validate.** `platform.core.get_index_mapping` on the target index to verify field existence/types → `ad_validate_job_spec`. If errors, fix and re-validate (max 3 attempts). 5. **Present and confirm.** Show the **complete** job + datafeed bodies formatted as the exact API calls. Ask for approval **once**. If feedback, incorporate and re-present (up to 3 rounds). 6. **Deploy.** After confirmation: `ad_create_job` → `ad_create_datafeed` → `ad_open_job` → `ad_manage_datafeed` (`action=_start`). Report final `job_id` and `datafeed_id`. For **batch analysis on historical data**, pass `start` and `end` to the datafeed start call. > Worked examples (rare-username, DNS exfil, large-downloads) with full JSON bodies and datafeed filters: > [references/job-creation-recipes.md](references/job-creation-recipes.md). ### Rules 1. **Create job before datafeed.** Datafeed references job by ID. 2. **Open job before starting datafeed.** Start on a closed job is rejected. 3. **`query_delay` = P95 ingest latency + buffer** (60s–120s safe default). 4. **Forecasts require non-population jobs** — `over_field_name` jobs cannot be forecasted; warn before attempting. 5. **`by_field_name` vs `over_field_name`:** `by` compares entity to its own history; `over` compares to peer group in the same bucket. `partition_field_name` = fully independent sub-model with its own normalization. 6. **`bucket_span` matches detection granularity** — 15m for high-frequency, 1h for operational metrics, 1d for daily patterns. Larger smooths short spikes; smaller increases noise. --- ## Registration (Kibana Agent Builder) Requires Node.js 18+. Defaults to `elastic`/`changeme` when no credentials are supplied. ```bash cd skills/kibana/kibana-anomaly-detection # tools → workflows → skills node scripts/kibana-agent-builder.mjs all register --kibana-url http://localhost:5601 # HTTPS with self-signed cert node scripts/kibana-agent-builder.mjs all register --kibana-url https://localhost:5601 --insecure ``` `all register` runs `tools register`, then `workflows register`, then `skills register`. Kibana allows **at most five** `tool_ids` per skill; the script fills them by scanning `SKILL.md` for tool mentions (in document order), then appends ids from `references/kibana/tools/esql/*.json` until the cap (workflow-only tools omitted by default). If you run `skills register` alone, run `tools register` first so those ids exist. Workflow tool exclusions and prefixes live in `scripts/agent_builder_constants.json`. **MCP API key permissions:** - Kibana: `read_onechat`, `space_read` - Index: `read`, `view_index_metadata` on `.ml-anomalies-*`, `.ml-annotations-*`, `.ml-notifications-*`, `.ml-config` - For source evidence: `read` on source data indices --- ## Tool inventory ES|QL tool specs live under `references/kibana/tools/esql/*.json`; workflow definitions under `references/kibana/workflows/*.yaml`. Each Mode section above lists the tools it uses. Full surface: [references/tools.md](references/tools.md) (ES|QL) and [references/workflow-tools.md](references/workflow-tools.md) (workflows). ### Key system indices | Index | Relevant content | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | `.ml-anomalies-*` | `record`, `bucket`, `influencer`, `model_plot`, `model_forecast`, `model_snapshot`, `category_definition`, `model_size_stats` | | `.ml-config` | job/datafeed documents (visible even for never-run jobs) | | `.ml-annotations-*` | delayed data (`event == "delayed_data"`) | | `.ml-notifications-*` | job messages (`level`: info/warning/error) | --- ## Examples **RCA:** "Something caused a spike in our error rate at 2pm — what broke?" → Investigate → `ad_get_available_metadata` → `ad_query_anomaly_timeline` → `ad_rca_cross_job_entity_match` → `ad_rca_multi_job_entities` → RCA report. **Score drop:** "My anomaly score went from 90 to 55 — did the model change?" → Explain → `ad_rca_score_reassessment` for drift → explain renormalization if `score_drift` is large. **Memory limit:** "Job status shows `hard_limit` and results look wrong." → Troubleshoot → `ad_ts_model_memory_health` → `ad_wf_ts_field_cardinality` → `ad_estimate_memory_requirement` → `ad_update_model_memory_limit` (lifecycle: stop datafeed → close → update → open → start). **New job:** "Detect unusual error rates per host on nginx access logs." → Manage → `high_count` detector with `by_field_name: "host.keyword"` → validate → present → deploy. **Multi-mode:** "We had an incident last night, scores were high but now low — is the job healthy?" → Investigate the incident → Explain the score drift → Troubleshoot if `hard_limit` or delayed data is suspected. --- ## Guidelines 1. **Pick a mode first.** Don't blend RCA logic with score-explanation logic in one response. 2. **`ad_validate_ml_tool_permissions` first** on empty results — privileges are the most common false-negative cause. 3. **Score bands are absolute thresholds**: `>75` critical, `50–75` warning, `25–50` minor, `<25` informational. 4. **Multi-job entities are prime suspects.** Use `min_job_count=2` in `ad_rca_multi_job_entities`. 5. **Show `initial_record_score` alongside `record_score`** — the gap tells the renormalization story. 6. **Fix memory before `query_delay`.** `hard_limit` invalidates downstream diagnostics. 7. **Stop datafeed → close job → update config → open job → start datafeed** for any config change to memory or query delay. 8. **Confirm RCAs with `ad_rca_source_evidence`.** Raw source documents are ground truth.