---
name: signoz-generating-queries
description: >
  Generate, write, or run an ad-hoc query against SigNoz observability
  data (metrics, logs, or traces) without wrapping it in
  a dashboard panel or alert. Make sure to use this skill whenever the
  user asks "show me error rates", "query logs for timeout errors",
  "what's the p99 latency for the cart service", "how many requests hit
  the payment endpoint", "find slow traces", "errors in the last hour",
  or otherwise asks an exploratory question that needs live observability
  data, even if they don't say "query" or "search" explicitly.
---

# Query Generate

## Prerequisites

This skill calls SigNoz MCP server tools heavily (`signoz_execute_builder_query`,
`signoz_query_metrics`, `signoz_search_logs`, `signoz_search_traces`,
`signoz_aggregate_logs`, `signoz_aggregate_traces`, `signoz_get_field_keys`,
`signoz_get_field_values`, `signoz_list_metrics`, `signoz_list_services`,
`signoz_get_service_top_operations`, `signoz_get_trace_details`). Before
running the workflow, confirm the `signoz_*` tools are available. If they
are not, run `signoz-mcp-setup` first to initialize or repair the MCP connection.
Do not fall back to raw HTTP calls or fabricate query results without the MCP
tools.

## When to use

Use this skill when the user asks to:
- Query, search, or look up observability data (traces, logs, metrics)
- Compute aggregations (error rate, p99 latency, request count, throughput)
- Find specific log entries, traces, or metric values
- Investigate patterns (spikes, drops, trends over time)

Do NOT use when:
- User wants raw ClickHouse SQL for a dashboard panel (custom joins, window
  functions, regex over log bodies); that's a separate dashboard-panel SQL
  workflow, not this skill.
- User needs the Exceptions Explorer as an exception-specific signal: the
  current MCP contract has no exception query tool. Explain that limitation and
  offer an equivalent error-trace or error-log query only if it answers their
  underlying question.

## Instructions

### Step 1: Determine the signal type

Map the user's intent to the right signal:

| User intent | Signal | Why |
|---|---|---|
| Error rate, latency, throughput, request count | **metrics** (preferred) or **traces** | Metrics are pre-aggregated and fastest. Use traces if the user needs per-request detail or no matching metric exists. |
| p50/p75/p90/p95/p99 latency | **metrics** (histogram) or **traces** (aggregate on `duration_nano`) | Prefer metrics if a histogram metric exists (e.g., `signoz_latency_bucket`). Fall back to trace aggregation. |
| Find specific log entries, error messages, stack traces | **logs** | Text search, pattern matching, severity filtering. |
| Find specific traces, slow requests, error spans | **traces** | Per-request detail, span attributes, duration filtering. |
| Infrastructure metrics (CPU, memory, disk, network) | **metrics** | Always metrics for resource utilization. |
| Ingestion volume (bytes or count), cost, or billing usage | **metrics** with `source=meter` (Cost Meter) | `signoz.meter.*` ingestion metrics (logs/spans/datapoints by count **and** bytes) live only in the meter store; bytes are unavailable on the raw signals. Dollar **cost is not a metric**; derive it from volume × per-unit price (see Step 2). groupBy/filter work like a normal metric, but only over the limited attribute set the meter retains (not arbitrary log/trace fields). For a *count* sliced by an attribute the meter doesn't carry, aggregate logs/traces directly instead. |
| "How many X per Y" (count/rate grouped by dimension) | **traces** or **logs** (aggregate) | Use `signoz_aggregate_traces` or `signoz_aggregate_logs` for grouped counts. |

Traces have no unqualified full-text search; use `CONTAINS` against a discovered
structured trace field. `searchText` convenience search is logs-only.

If the signal is genuinely ambiguous, ask the user before proceeding. The
host application decides how the question is surfaced (e.g. a structured
clarification tool or an inline `<assistant_question>` tag); follow the
host's UI rendering rules.

### Step 2: Discover available data

**Always discover before querying.** Use only names returned by tools, never
guess from training knowledge.

Run discovery calls in parallel where possible:

- **For metrics**: Call `signoz_list_metrics` with a `searchText` substring
  matching the user's intent (e.g., `searchText: "http"`, `searchText: "latency"`).
  The response includes metric type, temporality, and isMonotonic; pass these to
  `signoz_query_metrics` to avoid extra lookups. Before filtering or grouping,
  discover labels with `signoz_get_field_keys(signal: "metrics", metricName:
  <discovered-name>)`; metric labels are not trace/log attributes (`db.system`,
  for example, is a span attribute unless discovery also returns it for metrics).

  Choose `timeAggregation` from the discovered metric metadata:

  | Metric type | Valid `timeAggregation` |
  |---|---|
  | gauge | `latest`, `sum`, `avg`, `min`, `max`, `count`, `count_distinct` |
  | monotonic counter/sum | `rate`, `increase` |
  | non-monotonic sum | `avg`, `sum`, `min`, `max`, `count`, `count_distinct`; never `latest` or `rate` |
  | histogram / exponential histogram | omit; aggregation is automatic |
- **For Cost Meter** (ingestion volume, cost, billing): pass `source=meter` to
  `signoz_list_metrics` to discover the metrics (`signoz.meter.*`); they're
  invisible in the default store and the set evolves, so don't hardcode it.
  groupBy/filters/aggregations then work like any metric, with three caveats:
  *bytes exist only here* (count is also available via direct
  `signoz_aggregate_logs`/`_traces`); the meter retains only a *limited
  attribute set*, so discover groupable keys via `signoz_get_field_keys(signal:
  "metrics", source: "meter")`, and fall back to a direct count (no bytes) to
  slice by an attribute it lacks; and **dollar cost is not a meter metric**:
  the store holds only volume, so don't `searchText: "cost"` expecting a hit.
  For a cost question, query the volume metric (bytes for logs/traces, count
  for metric datapoints) and multiply by the per-unit price from Settings →
  Billing; ask the user for the price if you don't have it.
- **For traces**: Call `signoz_list_services` to confirm the service name exists,
  following `pagination.nextOffset` while `pagination.hasMore` is true before
  declaring it missing.
  Optionally call `signoz_get_service_top_operations` for the service to find
  operation names. Before any attribute filter or grouping, call
  `signoz_get_field_keys(signal: "traces")`.
- **For logs**: Before any attribute filter or grouping, call
  `signoz_get_field_keys(signal: "logs")`; use `signoz_get_field_values` only
  after discovery to validate values.

Field keys are signal-specific: `service.name` may be absent on logs, while
`body` is logs-only. Never carry a key across signals. Skip key discovery only
when live output in this conversation returned it for the same signal, never
for user text or memory. Use returned dot notation verbatim, not camelCase
guesses such as `traceID` or `hostname`. Never assume tenant keys (`org.id`,
`orgId`, `organization_id`, etc.); discover the actual key per signal. Call
`signoz_get_field_keys` before `signoz_get_field_values`; both require `signal`,
and values also require a non-empty discovered `name`.

### Step 3: Choose the right tool

**Use the simplest tool that answers the question:**

| Question type | Tool | When to use |
|---|---|---|
| Metric time series, scalar, ratio, or formula | `signoz_query_metrics` | Ordinary metrics queries, Cost Meter trends/rates, and metric ratios via `formula` + `formulaQueries`. |
| Cost Meter total or grouped total attribution | `signoz_execute_builder_query` | Use the discovered meter metric with raw `timeAggregation: "sum"`; sum complete hourly buckets. Do not use `signoz_query_metrics` for totals. |
| Log search (find matching entries) | `signoz_search_logs` | Finding specific log lines. Use `searchText` for literal text (`searchScope` chooses where it matches; the default is the log body), `filter` for field filters, `severity` for level filtering. |
| Trace search (find matching spans) | `signoz_search_traces` | Finding specific traces/spans. Use `service`, `operation`, `error`, `minDuration`/`maxDuration` shortcuts plus `filter` for field filters. |
| Log aggregation (count, avg, percentiles) | `signoz_aggregate_logs` | Plain totals, grouped counts, and top-N by one aggregation. |
| Trace aggregation (count, avg, percentiles) | `signoz_aggregate_traces` | Plain totals, grouped counts, and top-N by one aggregation. |
| Complex log/trace formula or shaping | `signoz_execute_builder_query` | Use when ratios, shaping, ordering, or cardinality control exceed the convenience tools; never hand-build a plain grouped count. |

**Canonical log search contract:**

- `signoz_search_logs` accepts `filter`; the former `query` alias is
  rejected.
- `searchText` is literal text to find, escaped automatically.
  `searchScope` chooses where it matches; the default is the log body. It
  combines with `filter` using AND. `searchScope` without `searchText` is a
  validation error.
- `searchScope` values: `body` (default) for message text; `attribute` or
  `resource` for their keys and values; `all` for every field. Scopes other
  than `body` are slower, and SigNoz may reject them over wide time ranges.
  Use them only when you don't know which key holds the text; when you do,
  put a predicate such as `attribute.<key> CONTAINS 'text'` in `filter`
  instead.
- For explicit cross-field search, use `filter: "search('timeout')"`.
  The `search()` scopes documented by `signoz://logs/query-builder-guide`
  support body, attribute, resource, log, or their allowed combinations.
  Unscoped `search()` means all supported log fields; enumerate scopes when
  the user requests only a subset.
- Prefer a direct known-field predicate such as
  `service.name = 'payments'` when the field is known, or use the `service`
  shortcut; the server composes that shortcut into the same ANDed
  `service.name` predicate. Discover unfamiliar fields first.
- When constructing a search literal, escape backslashes first and apostrophes
  second. Leave a caller-authored raw `filter` untouched; do not escape it
  again.

For `signoz_aggregate_logs` / `signoz_aggregate_traces`, `aggregation` is one
bare token: `count`, `count_distinct`, `avg`, `sum`, `min`, `max`, `p50`, `p75`,
`p90`, `p95`, `p99`, or `rate`. Never include parentheses or a column; put the
column in `aggregateOn` (`count` and `rate` need none). Omit `orderBy` for the
default aggregation-expression descending order; explicit ordering should use a
`groupBy` key or the aggregation expression. Custom aliases or formulas require
`signoz_execute_builder_query`.

For every `signoz_execute_builder_query` payload, put a positive `limit` and
non-empty Query Builder v5 `order` on each `builder_query` and
`builder_formula` spec. Raw requests and trace-signal `requestType: trace`
default to 100 rows: traces order by timestamp desc, while raw logs order by
timestamp desc then id desc. Scalar/time-series requests default to 100 groups
ordered by `__result` desc for metrics/formulas or the primary aggregation desc
for logs/traces. Formula results stay at 100, but every `builder_query`
referenced by a formula uses 10000 because each component limit is applied
before formula evaluation; independently top-100 inputs can drop a high-ratio
group. When calculating those input bounds, inspect every formula expression,
including formulas with `disabled: true`, and follow formula references until
all `builder_query` leaves are found. This dependency walk sets bounds only; it
does not prove deterministic formula-to-formula evaluation order, so validate
the complete composite payload. Narrow filters/grouping if input cardinality
can exceed 10000. This wire field is `order`, not dashboard editor `orderBy`.
Time-series top-N ranks groups over the whole window, so a short-lived local
spike may fall outside the returned set.

**`requestType` decision for aggregations:**
- `scalar` (default): "How many?", "What is the p99?", "Which service has the most?"
- `time_series`: "When did errors spike?", "How did latency change?", "Show trend"
- If the question has ANY temporal component (spike, trend, change), use `time_series`

Aggregate tools accept only `scalar` or `time_series`. For builder-query
envelopes passed to `signoz_execute_builder_query`, use only:

| Signal | Valid `requestType` |
|---|---|
| metrics | `time_series`, `scalar` |
| traces | `raw`, `trace`, `scalar`, `time_series` |
| logs | `raw`, `scalar`, `time_series` |

Never invent `aggregate`, `table`, `timeseries`, or `series`. This matrix does
not apply to PromQL or ClickHouse SQL envelopes.

### Step 4: Execute the query

- Always include `searchContext` with the user's original question; it improves
  result relevance.
- Respect the requested range; otherwise use 1h. Search/aggregate log and trace
  tools accept relative `timeRange` strings (`"1h"`, `"24h"`; default `1h`);
  prefer them. Valid Unix-ms `start`/`end` override `timeRange`; malformed explicit
  timestamps return validation guidance pointing to `timeRange`.
  `signoz_execute_builder_query` has no relative option: its outer `query` requires
  absolute `start` and `end` as JSON integer Unix-ms or fails with
  `missing start or end timestamp`.
- Use shortcut parameters (`service`, `severity`, `operation`, `error`) when they
  match the user's filters; they are simpler and less error-prone than building
  `filter` expressions.
- Combine shortcut params with `filter` for additional constraints; they
  are ANDed together.
- For `signoz_query_metrics`, pass `metricType`, `temporality`, and `isMonotonic`
  from the `signoz_list_metrics` response to avoid an extra auto-fetch round trip.
- For a **Cost Meter total or grouped total attribution**, use
  `signoz_execute_builder_query`, not `signoz_query_metrics`. Pass the full tool arguments with
  an outer `query` object, Unix-millisecond `start`/`end`, `requestType: "time_series"`,
  `formatOptions`, and `variables`. The builder spec keeps `signal: "metrics"`,
  `source: "meter"`, the discovered `metricName` and `temporality`, `stepInterval: 3600`,
  and raw `timeAggregation: "sum"` plus `spaceAggregation: "sum"`. For grouped totals,
  discover the meter field with `signoz_get_field_keys` and copy its `name`,
  `fieldDataType`, `fieldContext`, and `signal` into `groupBy` without dropping
  or translating fields. Exclude datapoints marked `partial: true`, then sum the
  complete hourly buckets for each returned group.
- `signoz_query_metrics` remains appropriate for a Cost Meter trend or rate that is not a total
  attribution. Carry `source=meter`, use `stepInterval: 3600`, use
  `timeAggregation: increase` for a volume trend, and `rate` only for a per-second rate.

### Step 5: Handle results

**Scope every claim to what the read returned.** Before reporting, classify
each result:

1. **Failed read**: the tool returned an error or timed out. This is not
   evidence of absence. Say the read failed, quote the error, and never report
   zero, "none", or a total from it.
2. **Partial result**: a raw search whose note reports `hasMore=true`, any
   response with a `nextCursor` or a note saying more results exist or may
   exist, or a `groupBy` aggregate that returned exactly `limit` groups
   (aggregate tools do not flag this, so treat it as truncated). Claims cover
   only the returned rows ("the 100 most recent matching logs all come from
   pod-a"). Never present a count of returned rows as a total, and never
   claim "none" or "all" for the full scope or window from a partial result.
3. **Complete result**: a successful aggregate over the full requested scope
   and window, or a raw search whose note reports `hasMore=false` (fewer rows
   than `limit`, including zero). Only this supports "none", "all", or a
   total over the full scope. For a total larger than one page, use
   `signoz_aggregate_logs` or `signoz_aggregate_traces` with
   `aggregation: count`.

**Data returned:**
- Present findings as neutral observations with timestamps and values.
- Include the time range in your response.
- For aggregations with `groupBy`, highlight the top entries. If exactly
  `limit` groups came back, say more groups may exist; the total group count
  is not reported.
- For search results, summarize patterns rather than listing every entry.

**No data returned.** Report what was empty and its scope, and claim nothing
beyond it:

1. **Zero over the requested scope**: a successful, complete query returned no
   matches. Say so with the filter and window: "No error spans matched
   `service.name = 'checkout-service'` in the last hour." Absence of errors or
   traces does not establish that the service is healthy, that it was never
   running, or why the data is missing. Offer such explanations only as
   labeled possibilities.
2. **No data in range**: the field or metric was seen in discovery, but no
   data points fall in the requested window. Report the empty window and
   suggest a wider range. Older data seen in a wider window is outside the
   requested window, not recent activity.
3. **Not found in discovery**: the metric, field, or service is absent from
   discovery results. Discovery is scoped and capped, and a miss in a
   truncated list shows nothing:
   - `signoz_list_services` lists only services with traces (6h by default).
     Follow `pagination.nextOffset` until `hasMore=false`. A service missing
     here may still send logs or metrics.
   - `signoz_list_metrics` lists metrics active in the window (1h by default)
     and has no paging. If its note reports `hasMore=true`, narrow with
     `searchText`.
   - `signoz_get_field_keys` and `signoz_get_field_values` set
     `data.complete=false` when they truncate. Narrow with `searchText`.

   From a complete list, report "not seen in <signal> for <window>" rather
   than "does not exist" or "not instrumented". Missing instrumentation is
   one possibility to check, alongside a wider window or a different name.

**Drill-down:**
- If an aggregation reveals an interesting pattern (spike, outlier service),
  offer to drill into individual traces or logs for that scope.
- If a trace search returns interesting spans, offer to fetch full trace details
  via `signoz_get_trace_details`.

## Guardrails

- **Discovery first**: Never guess metric, field, or service names. A field is
  confirmed only by live output for the same signal in this conversation.
- **Supply required parameters**: Never invoke tools hoping they auto-discover.
  `signoz_aggregate_*` requires `aggregation`; `signoz_query_metrics` requires a
  `metricName` from `signoz_list_metrics`; `signoz_get_trace_details` requires a
  `traceId`; `signoz_get_service_top_operations` requires a `service` from
  `signoz_list_services`. Discover first, then call.
- **Never claim root cause**: Present data patterns and correlations. Write
  "Error rate for checkout increased from 0.2% to 4.1% at 14:05" not "The
  deployment caused the errors."
- **Never claim absence beyond the evidence**: A failed, timed-out, or partial
  read shows nothing about absence. "None", "all", and totals need a complete
  result over the requested scope (Step 5).
- **One focused query per question**: Do not scatter-shot multiple queries when
  one precise query answers the question. Use parallel discovery calls, but be
  precise for execution.
- **Respect MCP server rules**: The MCP server enforces rules about resource
  attribute filters, filter operators, and redundant queries. Follow them,
  especially preferring resource attributes in filters for faster queries.
- **No raw ClickHouse SQL**: Always use the Query Builder tools. Never construct
  raw SQL.
- **Scope boundary**: This skill queries data. If the user wants to wrap the
  query into a recurring alert, redirect to `signoz-creating-alerts`.
- **Emit `apply_filter` on the final message.** When the user asks you to
  write, build, generate, or show a query, include an `apply_filter` action
  on your final assistant message with the exact full v5 `query` object you
  passed to a successful `signoz_execute_builder_query` call in this turn. The
  chip carries the entire query-range envelope (`schemaVersion: "v1"`, `start`,
  `end`, `requestType`, `compositeQuery`), not just the inner
  `compositeQuery`, and you must copy it verbatim rather than reconstructing
  it. If you answered via simplified tools (`signoz_search_logs`,
  `signoz_search_traces`, `signoz_aggregate_*`, `signoz_query_metrics`), run
  one validating `signoz_execute_builder_query` with a small `limit` and copy
  that exact query object, or skip the chip. Use the appropriate `signal`
  field (`metrics`, `logs`, or `traces`). This signals to the SigNoz UI that
  the user wants to apply the query to an explorer page. Only emit
  `apply_filter` when the user's primary intent is to obtain a runnable query,
  not when the user is asking a one-shot data question that the analysis text
  already answers. For a Cost Meter query keep `signal: metrics` and ensure the
  copied query spec carries `source: meter`.

## Examples

**User:** "Show me the error rate for the checkout service in the last hour"

**Agent:**
1. Calls `signoz_list_metrics(searchText: "calls", timeRange: "1h")` and
   selects the exact returned request-count metric. In this example discovery
   returns `signoz_calls_total` with `metricType: "sum"`,
   `temporality: "cumulative"`, and `isMonotonic: true`.
2. Calls `signoz_get_field_keys(signal: "metrics", metricName:
   "signoz_calls_total")` to discover both service and error labels, then
   `signoz_get_field_values` for each returned `name` / `fieldContext` with the
   same metric, confirming `checkout-service` and `STATUS_CODE_ERROR`. This
   example's returned names are `service.name` and `status_code`; use whatever
   names live discovery returns.
3. Calls `signoz_query_metrics` with these complete arguments. The primary
   arguments define query A; `formulaQueries` explicitly defines query B with
   a different filter so the denominator includes every checkout request. Both
   filters use the label names returned in Step 2 verbatim:

   ```json
   {
     "searchContext": "Show me the error rate for the checkout service in the last hour",
     "metricName": "signoz_calls_total",
     "metricType": "sum",
     "isMonotonic": true,
     "temporality": "cumulative",
     "timeAggregation": "increase",
     "spaceAggregation": "sum",
     "filter": "service.name = 'checkout-service' AND status_code = 'STATUS_CODE_ERROR'",
     "timeRange": "1h",
     "requestType": "scalar",
     "formula": "A / B * 100",
     "formulaQueries": [{
       "name": "B",
       "metricName": "signoz_calls_total",
       "metricType": "sum",
       "isMonotonic": true,
       "temporality": "cumulative",
       "timeAggregation": "increase",
       "spaceAggregation": "sum",
       "filter": "service.name = 'checkout-service'"
     }]
   }
   ```

   Replace every concrete metric, metadata value, field, and field value above
   with the tenant's discovery results; never assume this example exists.
4. Presents: "Error rate for checkout-service: 2.3% over the last hour (14:00–
   15:00 UTC). 47 errors out of 2,041 total requests."
5. Offers drill-down: "Want me to check which operations have the highest error
   rate?"

---

**User:** "Find timeout errors in logs"

**Agent:**
1. Calls `signoz_search_logs(searchText: "timeout", severity: "ERROR",
   timeRange: "1h")`.
2. Presents: "Found 23 timeout errors in the last hour. Most are from
   payment-service (18) and gateway (5). Common pattern: 'context deadline
   exceeded' in gRPC calls to downstream-api."
3. Offers: "Want me to check traces for these services to see where the
   timeouts occur?"

---

**User:** "What's the p99 latency for the cart service?"

**Agent:**
1. Calls `signoz_aggregate_traces(aggregation: "p99",
   aggregateOn: "duration_nano", service: "cart-service",
   requestType: "scalar", timeRange: "1h")`.
2. Presents: "p99 latency for cart-service: 1.2s over the last hour."
3. Offers: "Want me to break this down by operation or show the trend over time?"

---

**User:** "When did errors spike for the frontend?"

**Agent:**
1. Calls `signoz_aggregate_traces(aggregation: "count", error: "true",
   service: "frontend", requestType: "time_series", timeRange: "6h")`.
2. Presents: "Error count for frontend over the last 6 hours. Spike at 11:30 UTC:
   error count jumped from ~5/min to ~45/min, returning to baseline by 12:15."
3. Offers: "Want me to check what error types appeared during the spike?"

---

**User:** "How much log data is each service ingesting?"

**Agent:**
1. Bytes by service → Cost Meter. `signoz_list_metrics(searchText: "log",
   source: "meter")` and selects the returned log-volume metric from its live name and unit.
2. Calls `signoz_execute_builder_query` with the outer `query` wrapper,
   `schemaVersion: "v1"`, integer Unix-ms range,
   `requestType: "time_series"`, `formatOptions`, `variables`, and a builder spec using
   `signal: "metrics"`, `source: "meter"`, the discovered metric name and temporality,
   `stepInterval: 3600`, raw `timeAggregation: "sum"`, `spaceAggregation: "sum"`, and
   the `service.name` field returned by `signoz_get_field_keys`, copying its
   `name`, `fieldDataType`, `fieldContext`, and `signal` into `groupBy`.
3. Excludes `partial: true` datapoints, sums complete hourly buckets per service, and presents
   per-service ingestion in the discovered unit. (If the meter lacks a requested attribute,
   fall back to a direct count and note that bytes remain meter-only.)