--- name: observability-csp-investigation description: > Investigate cloud-provider (CSP) service issues — AWS today (the reasoning applies to any AWS service; schema and failure signatures are deepest for RDS, Lambda, SQS, ALB, EC2), and in future GCP and Azure — using OTel-schema cloud metrics in Elastic. Use when diagnosing cloud-resource problems, whether alert-triggered or ad hoc: database connection/CPU/memory pressure, function error rates and throttling, queue backlog and consumer lag, load-balancer 5xx, instance saturation. Correlates the resource under investigation against its own baseline, classifies the failure mode, and rules out co-occurring but unrelated conditions. compatibility: > Requires the `elastic` CLI (>= 0.2) with an Elasticsearch context, and cloud-provider metrics ingested through EDOT (the OpenTelemetry `awscloudwatch` receiver) into `metrics-aws..otel-*` data streams. The base floor is Elasticsearch 8.11 or later, or Serverless; every query is ES|QL. Alert-state and SLO lookups additionally need a Kibana context on the same deployment. Read-only. metadata: author: elastic version: 0.2.0 universal: true --- # Cloud Provider (CSP) Investigation Diagnose cloud-provider service issues from OTel-schema metrics and logs in Elastic. The guidelines, flow, and synthesis below are provider-agnostic and apply to every investigation. The per-provider **reference** carries the schema facts and failure-mode signatures that cannot be guessed — read it first. This is the CSP **domain layer**: which index holds a resource's telemetry, what shape its metrics take, and how to read them into a diagnosis. It assumes the harness can already author ES|QL — it does not teach query syntax, null handling, or function usage. Keep that split: this skill tells you _what_ to query and _how to interpret_ it; writing the query itself is the harness's job. ## Environment Configuration This skill executes Elasticsearch operations through the `elastic` CLI. If the [`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds. This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`, `GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API directly. ### Analysis without cluster access The CLI check above gates _querying the cluster_ — it does not gate analysis. When the user has already supplied the evidence in their question (metric values, alert payloads, log lines, dimension names), reason from that evidence and deliver the conclusion. When you genuinely do need data the user has not provided, still say what you would check and how — name the specific query, index, and field that would settle the question — and then ask for CLI setup. An answer that names the check is useful without a cluster; one that only asks for setup is not. Every ES|QL query in this skill and in the provider references runs via `POST /_query`. Discovering which data streams and fields a deployment actually has uses `GET /_resolve/index/` and `GET /_field_caps` (the way to learn whether a sparse counter has ever been mapped — see Guidelines). Active alerts are read with `GET kbn:/api/alerting/rules/_find`; shipped SLOs with `GET kbn:/api/observability/slos`. The [Operations](#operations) table maps each to its `elastic` CLI equivalent. Express predicates natively in ES|QL; do not use Query DSL or the `KQL` function. ## Router — read the provider reference first Identify the provider from the resource under investigation, then read its reference before writing any query (the index patterns, field shapes, and stat/aggregation rules there are unguessable, and a query written without them fails or silently corrupts its aggregates): | Provider | Resource types | Reference | | ---------- | --------------------------------------------------------------------------------------------------- | -------------------------------------- | | AWS | Any service under `metrics-aws.*`; failure signatures documented for RDS, Lambda, SQS, ALB/ELB, EC2 | [references/aws.md](references/aws.md) | | GCP, Azure | — | not yet covered (see below) | If the provider is unclear, determine it from the data before assuming: `GET /_resolve/index/metrics-*` lists the data streams that exist, and the provider is in their names. For a provider without a reference yet, apply the provider-agnostic guidance below, discover the schema empirically (list data streams → field mapping → probe query), cap confidence at medium, and say the reference was unavailable — never transplant another provider's schema. The same applies _within_ a provider. For an AWS service the reference doesn't detail (DynamoDB, ECS, API Gateway, Kinesis, ElastiCache, …), the general guidance and the AWS-wide conventions still hold — the `metrics-aws.*` index family, backticked dotted/slashed field names, and per-`attributes.stat` filtering — so apply those, discover that service's specific metrics empirically, and cap confidence where no documented failure-mode signature exists. The guidelines below are service-agnostic by design; only the tuned signatures are per-service. ## Guidelines **Investigate the named resource before anything else — the one the alert fired on, or the one the request names.** Shared clusters carry many systems' telemetry (Kubernetes workloads, other teams' accounts, demo apps). Signals from co-resident systems are not evidence about this alert unless a dependency between them is demonstrated. Do not promote a louder co-resident anomaly to root cause. **The alert's own metric, compared to its own baseline, outranks everything else.** Before considering any other metric, quantify the alerting metric's incident-window value against a representative window of the same resource's own normal (how to choose that window — trailing vs same-time-prior-day — is in Flow below; the point is it must be _representative_, not merely adjacent). A 10×+ delta on the alert's metric is the primary thread; small wiggles in other metrics are secondary until the primary thread is exhausted. "High" is only meaningful relative to this resource's normal. **Counters up ≠ failing harder. Always compute the rate.** For error counters, divide by the volume counter (errors/invocations, 5xx/requests) in BOTH windows. Volume up with rate flat = load change, not a failure. Rate up with volume flat = a real failure in the resource's own code/config path. **One instance degrading while its fleet siblings stay healthy = the cause is scoped to that instance.** Check the same metric on sibling resources. Healthy siblings rule out platform-wide issues, shared event sources, and shared downstream dependencies — the fault is in the affected instance's own code, configuration, or deployment. **Co-occurring incidents are not causally linked by timing alone.** Attribute cross-service causation only when (a) a dependency actually exists (the service calls that resource), and (b) the upstream degradation preceded the downstream symptom. Two resources alerting in the same window on a busy cluster is the norm, not a clue. When you notice another resource anomalous in the same window, TEST the dependency before linking: does the affected resource actually call it? A worker that only touches a queue cannot be broken by a database incident. If you cannot demonstrate the dependency from the data, report the other incident as concurrent-and-unrelated and keep diagnosing the named resource on its own signals. **A healthy verdict is a first-class outcome, not a failure to find the problem.** Many workloads run with an ambient error rate of several percent at all times. If the named resource's incident-window numbers are indistinguishable from its baseline (rate ratio near 1×; volume, duration, throttles, backlog at baseline), the correct conclusion is that the alert is spurious or already resolved (or, for an ad-hoc check with no alert, that there is no problem): state it plainly — "ALERT FIRED BUT SYSTEM APPEARS HEALTHY", or "NO PROBLEM FOUND — `` IS AT BASELINE" — show the incident-vs-baseline numbers, and stop. Constructing a root-cause story out of ambient fluctuation is a worse failure than reporting no incident. **Name exactly one primary cause; everything else is an effect, a contributor, or unrelated.** When one resource saturates (e.g. CPU pinned at 100%), the metrics it drives rise with it — IO, latency, queue depth. Those are downstream effects, not co-equal causes. A conclusion naming two "co-primary" bottlenecks is usually an unfinished diagnosis: determine which one explains the other and say so. **Logs carry the "why" that metrics cannot — cite them when available.** Metrics localize _which_ component failed and rule failure modes in or out; the specific cause (the exception, the failing query, the bad config) lives in the resource's logs. After classifying from metrics, check the implicated resource's logs and cite the concrete error — turn "the function's code path is failing" into "RuntimeError at index.py:11". Logs arrive in one of two shapes and you should check both: OTel-native `logs-*.otel-*` (filter by `service.name`, line in `body.text`) and the classic cloud-integration shape (e.g. AWS `logs-aws_logs.*`, line in the `message` field). Zero rows in one shape is not "no errors" — the logs may be in the other, or not shipped at all; say which you checked. Reading a log line does not excuse misattribution: if a function logs its own exception, that is the cause, even when another resource is alerting in the same window. **Absence of a metric field is data.** Cloud providers omit zero-valued sparse counters (error counts, per-code 5xx, throttle counts), so the field may be unmapped in Elastic until the first such event ever occurs — and ES|QL fails the whole query on an unknown column. Check the field with `GET /_field_caps` before referencing it, or query sparse fields in separate small queries; a verification failure or empty result on a sparse counter means "no such events ever recorded," never "data missing." **Report insufficient evidence rather than manufacturing a conclusion.** If the named resource has zero datapoints in the window, say so, list the exact index patterns and filters you tried, and stop. Recent windows may simply not be ingested yet (cloud-metric pipelines typically lag by minutes — the provider reference gives the number). ## Flow 1. **Orient: resolve the resource and the two windows.** The incident window brackets the time of interest — the alert time, or the reported symptom time. The baseline is a _representative_ span of the same resource's normal, chosen to match its rhythm: a short trailing window (~30–60m before the incident) is fine for a steady workload, but for anything with daily or weekly seasonality widen the window or compare against the same time-of-day on a prior comparable day — a half-hour adjacent baseline makes normal diurnal variation read as the incident. Windows must not overlap. When the resource's rhythm is unknown, look back far enough to see at least one full normal cycle before trusting "high". Decision: which data stream, which dimension value, which two time ranges every later query uses. Data: `GET /_resolve/index/` for the streams that exist; the alert payload or request for the resource name. 2. **Quantify the alert's own metric against its baseline.** One `POST /_query` per window (or one query with a per-window `CASE`), using the stat filter and backticked field names from the provider reference. Decision: is the delta material (roughly ≥2× the resource's own baseline rate, or 10×+ on an absolute metric), or is the resource at baseline? If at baseline, go to step 5 with a healthy verdict. 3. **Classify against the provider reference's failure-mode signatures.** Pull the discriminating metrics the signature table names (the ceiling plateau, the throttle counter, the deleted-vs-sent ratio, the target-vs-ELB 5xx split) and compute the rates. Decision: exactly one named mode, or "no documented signature matches" with confidence capped at medium. 4. **Corroborate.** Siblings (`POST /_query` for the same metric across the fleet); dependent or upstream resources only where a real dependency exists; the implicated resource's logs in both shapes; shipped SLOs (`GET kbn:/api/observability/slos`) and active alerting rules (`GET kbn:/api/alerting/rules/_find`, filtering client-side for rules whose params name this resource) to anchor severity. Decision: does anything contradict the classification? Anchor severity on a VIOLATED SLO or an active alert when present; never invent one. 5. **Synthesize and stop.** Terminate as soon as the evidence supports a classification at known confidence; exhaustive exploration is a failure mode. ## Synthesis State: root cause (one sentence, named mode + resource), evidence with concrete numbers (incident vs baseline), causal chain, scope (which resources affected, which ruled out and why), recommended action targeting the actual mode, and confidence. Downgrade confidence when a discriminating signal was unavailable and say which. "ALERT FIRED BUT THE SYSTEM APPEARS HEALTHY" (or "no problem found" for an ad-hoc check) and "insufficient evidence for the named resource" are valid, complete conclusions when the data supports them. ## Examples ### "The RDS connections alert fired on `orders-db` — what's wrong?" Orient on `metrics-aws.rds.otel-*` filtered by `attributes.DBInstanceIdentifier == "orders-db"`, incident window around the alert, baseline the same hour on the prior day (databases are diurnal). Quantify `DatabaseConnections` (stat `Maximum`) in both windows. If it climbed far above baseline and plateaued while CPU, memory, storage and IO stayed nominal, that is the connection-exhaustion signature — RDS emits no `max_connections`, so infer the ceiling from the plateau. Corroborate on sibling instances (healthy siblings scope the cause to this instance's callers), then pull the callers' logs for the concrete error (`too many connections`, pool timeouts). Recommend the fix for the mode: raise the ceiling or shrink the callers' pools, not a restart. ### "Lambda `order-worker` error rate jumped, and the database is alerting at the same time" Compute the rate, not the count: `Errors / Invocations` (stat `Sum`) in both windows. If the rate rose at flat volume, the function's own code path is failing. Before linking it to the database incident, test the dependency: a queue consumer that never calls the database cannot be broken by it. Pull the function's own logs (`logs-aws_logs.*` or `logs-*.otel-*`) and cite the exception. Report the database incident as concurrent-and-unrelated unless the logs show the function calling it and failing on that call. If volume rose and the rate did not, it is a load surge — nothing is broken. ### "Alert fired but everything looks healthy" A first-class outcome. Show the incident-vs-baseline numbers for the alerting metric and its volume counter; if the ratio is near 1× and the discriminating metrics are at baseline, state `ALERT FIRED BUT SYSTEM APPEARS HEALTHY`, list what you checked, and recommend tuning the rule (window, threshold relative to the resource's own normal) rather than constructing a cause from ambient fluctuation. ## Operations | HTTP API (shorthand) | `elastic` CLI command | | ----------------------------------- | ---------------------------------------------------------------------- | | `GET /` | `elastic es info` | | `POST /_query` | `elastic es esql query --format tsv --query ''` | | `GET /_resolve/index/` | `elastic es indices resolve-index --name ''` | | `GET //_mapping` | `elastic es indices get-mapping --index ''` | | `GET /_field_caps` | `elastic es field-caps --index '' --fields ''` | | `GET kbn:/api/alerting/rules/_find` | `elastic kb alerting get-alerting-rules-find --filter ''` | | `GET kbn:/api/observability/slos` | `elastic kb slo find-slos-op --space-id '' --kql-query ''` |