--- name: datadog-insights description: Investigate Gram production health and post a digest to Slack --- # Gram Production Health Digest You are producing a health report for Gram's production services. The report must be **actionable** and **visually structured** — critical issues must stand out immediately, tabular data must use code blocks, and every section must be separated by a divider. **Before starting**: activate the `datadog` skill for Gram service names, MCP tools, and query guidelines. > ⚠️ **MANDATORY FORMAT RULES — READ BEFORE COMPOSING THE MESSAGE:** > > 1. **Every major section MUST be preceded by a Unicode divider line**: `──────────────────────────────────────` on its own line, with a blank line above and below. > 2. **Top endpoints, error type breakdowns, and latency tables MUST use triple-backtick code blocks** — never bullet points for tabular data. > 3. **Code block tables must have aligned columns** using spaces. Minimum widths: endpoint 38 chars, count 8 chars, 4xx% 6 chars, 5xx% 6 chars, p95 8 chars. > 4. **Each monitor in alert MUST get its own paragraph** — never combine multiple monitors into one block. > 5. **Do NOT collapse or omit data** to save space. If there are 8 monitors, show all 8. --- ## Step 1: Check for critical issues first These take priority over everything else. If any exist, they become the top of the digest. 1. **Open incidents** — `search_datadog_incidents` for `state:(active OR stable)` in the last 24h 2. **Monitors in alert** — `search_datadog_monitors` with query `status:alert (notification:slack-Speakeasy-gram-oncall OR notification:slack-oncall-gram)`. This filters to Gram-only monitors. Never include monitors that don't notify one of these two channels. 3. **Error spikes** — Use `analyze_datadog_logs` with SQL: ```sql SELECT service, status, count(*) FROM logs GROUP BY service, status ORDER BY count(*) DESC ``` Filter: `env:prod status:(error OR critical OR alert OR emergency)`, last 24h. Compare the last 6h vs. the previous 18h to detect spikes. If there are critical issues, investigate each one: - Get a sample of the actual error logs (`search_datadog_logs`) - Follow trace IDs with `get_datadog_trace` to find root causes - `Grep` in `server/internal/` for the error message to find the source code location **For top error message breakdown**, use `analyze_datadog_logs`: ```sql SELECT message, count(*) as cnt FROM logs WHERE service = 'gram-server' AND status IN ('error', 'critical') GROUP BY message ORDER BY cnt DESC LIMIT 10 ``` --- ## Step 2: Top endpoints by traffic Use `search_datadog_spans` for `service:gram-server env:prod` over the last 24h, or: ``` sum:trace.http.server.request.hits{service:gram-server,env:prod} by {resource_name}.rollup(sum, 86400) ``` Collect the **top 10 endpoints** with: - Request count - 4xx count and rate (% of requests returning 4xx) - 5xx count and rate (% of requests returning 5xx) - p95 latency Keep 4xx and 5xx separate — never fold them into a single error rate. 4xx is mostly client behaviour (bad auth, missing resources) and is expected on public endpoints like `/mcp/{mcpSlug}`, while 5xx indicates a server fault. Use `get_datadog_metric` grouped by status code and bucket the series by leading digit: ``` sum:trace.http.server.request.hits{service:gram-server,env:prod} by {resource_name,http.status_code}.rollup(sum, 86400) ``` If the metric is missing the `http.status_code` tag, fall back to `aggregate_spans` over `service:gram-server env:prod` grouped by `resource_name`, once with `@http.status_code:[400 TO 499]` and once with `@http.status_code:[500 TO 599]`. --- ## Step 3: Traffic volume and trends Compare traffic between two 12h windows: 1. **Current 12h**: `from: now-12h, to: now` 2. **Previous 12h**: `from: now-24h, to: now-12h` Use `get_datadog_metric` with: ``` sum:trace.http.server.request.hits{service:gram-server,env:prod}.rollup(sum, 43200) ``` Report: - Total requests in the last 24h - % change between the two 12h periods (flag if > 30% change) - Per-service breakdown (`gram-server`, `gram-worker`, `gram`, `fly`) --- ## Step 4: Latency analysis ``` p50:trace.http.server.request{service:gram-server,env:prod} by {resource_name} p95:trace.http.server.request{service:gram-server,env:prod} by {resource_name} p99:trace.http.server.request{service:gram-server,env:prod} by {resource_name} ``` Over the last 24h with `.rollup(avg, 86400)`. Report: - **Global latency**: p50, p95, p99 across all endpoints - **Slowest 5 endpoints** by p95 latency (with their p50 for comparison) - Flag any endpoint where p95 > 2s or p99 > 5s --- ## Step 5: Create a Datadog Notebook Call `create_datadog_notebook` with name `"Gram Health Digest — "` (e.g. `"Gram Health Digest — Fri 2026-03-27"`). Use `absolute_time: true` with `start_time` = 24h ago and `end_time` = now. One notebook is created per run — old ones accumulate and can be manually deleted periodically. The notebook `cells` must be wrapped in `{"cells": [...]}`. Include: 1. **Summary markdown cell**: ```json { "type": "notebook_cells", "attributes": { "definition": { "type": "markdown", "text": "One paragraph verdict with key numbers." } } } ``` 2. **Error rate timeseries cell**: ```json { "type": "notebook_cells", "attributes": { "definition": { "type": "timeseries", "title": "gram-server Error Rate (1h buckets)", "requests": [ { "q": "sum:trace.http.server.request.errors{service:gram-server,env:prod}.rollup(sum, 3600)", "display_type": "bars", "style": { "palette": "warm" } } ], "show_legend": true, "yaxis": { "scale": "linear" }, "markers": [ { "value": "y = 500", "display_type": "warning dashed", "label": "Elevated" } ] } } } ``` 3. **Traffic volume timeseries cell**: ```json { "type": "notebook_cells", "attributes": { "definition": { "type": "timeseries", "title": "gram-server Traffic Volume (1h buckets)", "requests": [ { "q": "sum:trace.http.server.request.hits{service:gram-server,env:prod}.rollup(sum, 3600)", "display_type": "area", "style": { "palette": "dog_classic" } } ], "show_legend": true, "yaxis": { "scale": "linear" } } } } ``` 4. **p95 latency by endpoint timeseries cell**: ```json { "type": "notebook_cells", "attributes": { "definition": { "type": "timeseries", "title": "Top Endpoint p95 Latency", "requests": [ { "q": "p95:trace.http.server.request{service:gram-server,env:prod} by {resource_name}.rollup(avg, 3600)", "display_type": "line", "style": { "palette": "dog_classic" } } ], "show_legend": true, "yaxis": { "scale": "linear" }, "markers": [ { "value": "y = 2", "display_type": "error dashed", "label": "2s threshold" } ] } } } ``` 5. **gram-worker error rate timeseries cell**: ```json { "type": "notebook_cells", "attributes": { "definition": { "type": "timeseries", "title": "gram-worker Error Rate (1h buckets)", "requests": [ { "q": "sum:trace.http.server.request.errors{service:gram-worker,env:prod}.rollup(sum, 3600)", "display_type": "bars", "style": { "palette": "warm" } } ], "show_legend": true, "yaxis": { "scale": "linear" } } } } ``` 6. **gram (frontend) trace errors timeseries cell** — `gram` is an APM service, so use trace metrics: ```json { "type": "notebook_cells", "attributes": { "definition": { "type": "timeseries", "title": "gram (frontend) Trace Errors (1h buckets)", "requests": [ { "q": "sum:trace.http.server.request.errors{service:gram,env:prod}.rollup(sum, 3600)", "display_type": "bars", "style": { "palette": "warm" } } ], "show_legend": true, "yaxis": { "scale": "linear" } } } } ``` 7. **fly (functions) error log stream cell** — `fly` is a log source (not an APM service), so use a log stream, not a trace metric: ```json { "type": "notebook_cells", "attributes": { "definition": { "type": "log_stream", "title": "fly (functions) Error Logs (24h)", "query": "source:fly env:prod status:error", "columns": ["timestamp", "host", "message"], "message_display": "inline", "show_date_column": true, "show_message_column": true, "sort": { "column": "timestamp", "order": "desc" } } } } ``` 8. **Slow endpoints + top errors markdown table cell** with the real data from Steps 1–4. 9. **All Gram services error log stream cell** — includes `source:fly` for Gram Functions logs: ```json { "type": "notebook_cells", "attributes": { "definition": { "type": "log_stream", "query": "(service:(gram-server OR gram-worker OR gram) OR source:fly) env:prod status:error", "columns": ["timestamp", "host", "service", "message"], "message_display": "inline", "show_date_column": true, "show_message_column": true, "sort": { "column": "timestamp", "order": "desc" } } } } ``` Save the notebook URL — you will link it in the Slack message footer. --- ## Step 6: Write a recommendation Based on all the data gathered, write **one concrete recommendation** for the on-call engineer. Be specific: - If errors are spiking: name the error type, the likely cause based on code grep, and the first action to take. - If a slow endpoint is flagged: name it and suggest where to look (query, external call, etc.). - If everything is healthy: say "No action needed. Monitor X for Y." - If errors are declining: say so and advise continued monitoring. This recommendation goes into the Slack message as a dedicated section. --- ## Step 7: Compose the Slack Block Kit message Build a list of Block Kit blocks. The message is structured around the **4 Golden Signals**: Alerts → Errors → Traffic → Latency. ### Formatting rules - **Prose and bullet lists**: use mrkdwn bullet points (`•`) with inline backtick formatting for endpoint/service names - **Tabular data** (error type breakdowns, endpoint tables, latency tables): use triple-backtick code blocks inside `section` mrkdwn text — they render as aligned monospace in Slack and are much more readable than bullet points for columnar data - **Verdict**: use a `section` with `fields` (2-column grid) — never a `context` block, which is too small to notice ### Verdict emoji rules - 🔴 if any active incidents or monitors in ALERT state - 🟡 if elevated error rates (>1.5x normal), notable latency, or monitors in WARNING/ALERT state - 🟢 if everything looks healthy --- ### Block structure (in order) **1. Header** ```json { "type": "header", "text": { "type": "plain_text", "text": "Gram Health Digest — " } } ``` **2. Verdict — `section` with `fields` (2-column grid)** Always 6 fields: Status, Monitors in Alert, Errors (24h), Traffic (24h), Latency p95, Slow Endpoints. ```json { "type": "section", "fields": [ { "type": "mrkdwn", "text": "*Status*\n " }, { "type": "mrkdwn", "text": "*Monitors in Alert*\n or 0 🟢" }, { "type": "mrkdwn", "text": "*Errors (24h)*\n · ↑ last 6h" }, { "type": "mrkdwn", "text": "*Traffic (24h)*\n~ · <↑/↓pct%> last 12h" }, { "type": "mrkdwn", "text": "*Latency p95*\n (global)" }, { "type": "mrkdwn", "text": "*Slow Endpoints*\n 2s> or All healthy 🟢" } ] } ``` Follow with a divider. --- **3. 🚨 Alerts** (omit section entirely if no monitors in alert) Each monitor gets its own paragraph. Do NOT combine monitors. ```json { "type": "section", "text": { "type": "mrkdwn", "text": "🚨 *Alerts*\n🔴 **\n\n*Notifying:* `#`\n\n🔴 **\n\n*Notifying:* `#`" } } ``` Follow with a divider. --- **4. ❌ Errors** Bullet prose for per-service summary, then a **code block table** for top error types. ```` {"type": "section", "text": {"type": "mrkdwn", "text": "❌ *Errors*\n• `gram-server`: X errors in last 6h (Y/h) vs Z/h prior — *~Nx spike*\n• `gram-worker`: N errors (stable)\n• `gram` (frontend): N (stable)\n• `fly` (functions): 0 🟢\n\n*Top error types — gram-server (24h):*\n```\nmessage count pct\nnot found 402 31.4%\ntoken value is empty for bearer auth 270 21.1%\nmissing value for env var in api key auth 74 5.8%\nHTTP roundtrip failed 70 5.5%\nno MCP install page metadata for toolset 65 5.1%\n```"}} ```` Follow with a divider. --- **5. 📊 Traffic** Bullet prose for trend, then a **code block table** for top endpoints by volume with separate 4xx and 5xx columns. Never merge 4xx and 5xx into one error column. Flag any endpoint with a 5xx rate > 1% with ⚠️. ```` {"type": "section", "text": {"type": "mrkdwn", "text": "📊 *Traffic*\n• Previous 12h: ~Xk requests\n• Current 12h: ~Xk requests — *↑Y%* ⚠️ (flag if >30%)\n• Total 24h: ~Xk · 4xx: N (X%) · 5xx: N (X%)\n\n*Top endpoints by volume (24h):*\n```\nendpoint hits 4xx 5xx\nPOST /mcp/{mcpSlug} 103,784 8.2% 0.1%\nPOST /rpc/hooks.otel/v1/logs 16,824 0.0% 0.0%\nPOST /rpc/hooks.claude 14,956 0.3% 0.0%\nGET /mcp/{mcpSlug} 14,454 2.1% 1.4% ⚠️\nGET /.well-known/oauth-protected-resource 6,789 0.0% 0.0%\n```"}} ```` Follow with a divider. --- **6. ⏱️ Latency** If any endpoint has p95 > 2s, use a **code block table** for slow endpoints. Always include "approaching threshold" if any endpoints are 1–2s p95. ```` {"type": "section", "text": {"type": "mrkdwn", "text": "⏱️ *Latency*\n*Global:* p50: Xms · p95: Xms · p99: Xms\n\n*Slow endpoints (p95 > 2s):*\n```\nendpoint p95 p50 hits\nGET /rpc/toolsets.listfororg 7,275ms 5,766ms 57 ⚠️\nGET /rpc/usage.getperiodusage 5,173ms 3,403ms 49 ⚠️\nPOST /chat/completions 4,713ms 2,615ms 15 (AI)\n```\n*Approaching threshold (p95 > 1s):*\n```\nGET /rpc/environments.list 1,406ms 57\nGET /rpc/access.listgrants 1,281ms 84\n```"}} ```` If all endpoints are fast: ```json { "type": "section", "text": { "type": "mrkdwn", "text": "⏱️ *Latency* — All endpoints healthy. p50: Xms · p95: Xms · p99: Xms 🟢" } } ``` Follow with a divider. --- **7. Recommendation** ```json { "type": "section", "text": { "type": "mrkdwn", "text": "💡 *Recommendation*\n" } } ``` Follow with a divider. --- **8. Footer** — links to Datadog notebook and skill source ```json { "type": "context", "elements": [ { "type": "mrkdwn", "text": "🔴 Critical 🟡 Warning 🟢 Healthy | | " } ] } ``` Replace `NOTEBOOK_URL` with the actual notebook URL from Step 5. --- ## Step 8: Post to Slack Write and run this Python script via Bash. Post to `#gram-datadog-insights` by default, unless a different channel was specified in the prompt. ```python import json, urllib.request, os, datetime now_utc = datetime.datetime.utcnow() digest_date = now_utc.strftime("%a %b %-d") # e.g. "Mon Apr 20" env_path = os.path.expanduser("~/.config/gram/.env") token = None with open(env_path) as f: for line in f: if line.startswith("SLACK_BOT_TOKEN="): token = line.split("=", 1)[1].strip().strip('"').strip("'") break if not token: raise RuntimeError("SLACK_BOT_TOKEN not found in ~/.config/gram/.env") channel = "C0AKLE930BX" # #gram-datadog-insights — override with channel name if specified in prompt blocks = [] # replace with actual Block Kit blocks from Step 7 — use f"Gram Health Digest — {digest_date}" in the header block def slack_post(payload): data = json.dumps(payload).encode() req = urllib.request.Request( "https://slack.com/api/chat.postMessage", data=data, headers={"Content-Type": "application/json", "Authorization": f"Bearer {token}"}, method="POST", ) with urllib.request.urlopen(req) as resp: return json.loads(resp.read()) result = slack_post({ "channel": channel, "text": "Gram Health Digest", "blocks": blocks, }) if not result.get("ok"): raise RuntimeError(f"Slack error: {result}") ts = result["ts"] reply = slack_post({ "channel": channel, "thread_ts": ts, "text": "", }) if not reply.get("ok"): raise RuntimeError(f"Thread reply error: {reply}") print(f"✓ Posted to {channel} (ts={ts}), oncall tagged in thread") ``` MANDATORY RULES — never violate: - Post ALL content as ONE main message. Never split the digest across multiple messages. - NEVER send test or placeholder messages. Only post if you have real data from Step 1. - The thread reply must contain ONLY the oncall tag — nothing else.