---
name: observe
description: Query and manage Observe using the Observe CLI. Use when the user wants to run OPAL queries, list datasets, manage objects, or interact with their Observe tenant from the command line.
---

You are an expert at using the Observe CLI (`observe`) to interact with Observe tenants. The binary is located at `~/go/bin/observe`.

## Setup & Authentication

Before running any commands, check if the user has a config file:

```
cat ~/.config/observe.yaml
```

If not configured, help them log in:

```
~/go/bin/observe --customerid <CUSTOMER_ID> --site observeinc.com login <EMAIL> --sso
```

Login methods:
- `--sso` (recommended): Opens a browser-based approval flow. Best for SSO/SAML users (Okta, Azure AD, Google, PingOne).
- `--read-password`: Reads password from stdin (no echo). Best for email/password users.
- Direct password on CLI: `observe login user@example.com mypassword` (less secure, avoid in interactive shells).

The login command saves the auth token to `~/.config/observe.yaml` under the active profile (default: `default`). Use `--no-save` to prevent saving.

## Configuration File

The config file at `~/.config/observe.yaml` supports multiple profiles:

```yaml
profile:
  default:
    customerid: "133742069123"
    site: "observeinc.com"
    authtoken: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  staging:
    customerid: "180316196377"
    site: "observe-staging.com"
    authtoken: "YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY"
    workspace: MyWorkspace
```

Select a profile with `--profile=<name>` or the `OBSERVE_PROFILE` environment variable.

## Running OPAL Queries

The primary command for querying data:

```
~/go/bin/observe query -q '<OPAL_QUERY>' -i '<INPUT_DATASET>'
```

### Key Flags

| Flag | Short | Description |
|------|-------|-------------|
| `--query` | `-q` | OPAL query text |
| `--input` | `-i` | Input datasets: ID or `Workspace.Folder/Name` (comma-separated for joins) |
| `--start-time` | `-s` | Start of query time window |
| `--end-time` | `-e` | End of query time window |
| `--relative` | `-r` | Duration of query window (e.g., `4h`, `30m`, `1d`) |
| `--csv` | `-c` | Output in CSV format |
| `--json` | `-j` | Output in nd-JSON format |
| `--extended` | `-x` | Print one column value per row (useful for long values) |
| `--col-width` | `-w` | Max column width for table format; 0 for unlimited (default: 64) |
| `--file` | `-f` | Read OPAL query from a file |
| `--literal-strings` | `-l` | Print embedded control characters literally |
| `--format` | | Specify output format: `table`, `extended`, `csv`, `ndjson` |

### Input Datasets

Inputs can be specified as:
- Dataset ID: `-i '41021818'`
- Workspace.Name: `-i 'Default.kubernetes/Container Logs'`
- Multiple inputs for joins: `-i '41021818,right=41012929'` or `-i 'Default.System,right=Default.network/Flow Logs'`

If the default workspace is set in config, you can omit the workspace prefix.

### Time Windows

Default: last hour, truncated to full minutes. Specify at most two of: `--start-time`, `--end-time`, `--relative`.

Time formats supported:
- RFC 3339: `2023-04-20T16:20:00Z`
- With timezone: `2023-04-20T16:20:00.123-08:00`
- Relative: `-1d`, `-4h`, `-30m`
- With snap: `-1d@1h` (1 day ago, snapped to start of hour)
- Epoch seconds: `1682007600`
- Epoch milliseconds: `1682007600000`
- Epoch nanoseconds: `1682007600000000000`

### Query Examples

Simple query:
```
~/go/bin/observe query -q 'limit 10' -i 'Default.System'
```

Query with time window and extended output:
```
~/go/bin/observe query -q 'pick_col timestamp, log | limit 10' -i 'Default.kubernetes/Container Logs' -r 4h -x
```

Query with join:
```
~/go/bin/observe query \
    -q 'leftjoin host_ip=@right.dst_ip, source:@right.src_ip' \
    -i '41021818,right=41012929' \
    -r 4h
```

Query from file with CSV output:
```
~/go/bin/observe query -f my_query.opal -i 'Default.System' --csv -r 24h
```

JSON output (useful for piping to jq):
```
~/go/bin/observe query -q 'limit 5' -i 'Default.System' --json
```

## OPAL Query Patterns

### Filtering
```opal
filter body ~ /error|fail|exception/i          # regex match (case insensitive)
filter service_name = "payment"                 # exact match
filter pod ~ /payment/                          # partial regex match
filter error = true                             # boolean filter
filter body ~ /error/i and pod ~ /payment/      # combined filters
```

### Aggregation with statsby
```opal
statsby count:count(1), group_by(body)                        # count by message
statsby count:count(1), group_by(pod)                         # count by pod
statsby count:count(1), group_by(loyalty_level, error)        # multi-dimensional grouping
```

### Time series with timechart
```opal
timechart 1h, count:count(1)                                  # hourly counts
timechart 5m, count:count(1)                                  # 5-minute buckets
filter error = true | timechart 1h, count:count(1)            # error rate over time
```

### Column selection and extraction
```opal
pick_col timestamp, body, pod, namespace                      # select specific columns
make_col loyalty_level: string(attributes["app.loyalty.level"])  # extract from JSON
make_col card_type: string(attributes["app.payment.card_type"])  # extract nested attribute
```

### Conditional logic
```opal
make_col failure_rate: case(body = "error msg", count, true, 0)
```

## Common Dataset Types

Observe workspaces typically contain these dataset categories:

### Kubernetes Logs
- **Columns**: `timestamp`, `body`, `stream`, `cluster`, `namespace`, `container`, `pod`, `node`, `attributes`, `resource_attributes`, `instrumentation_scope`, `fields`, `meta`
- **Use for**: Log search, error analysis, pod-level debugging

### Tracing / Spans
- **Columns**: `start_time`, `end_time`, `duration`, `service_name`, `span_name`, `error`, `response_status`, `status_code`, `status_message`, `span_type`, `attributes`, `resource_attributes`, `parent_span_id`, `span_id`, `trace_id`
- **Use for**: Distributed trace analysis, service dependency mapping, latency investigation
- **IMPORTANT**: When using `pick_col` on span datasets, you MUST always include `start_time` and `end_time` columns, or the query will fail with: `need to pick 'valid from' column "start_time"` / `need to pick 'valid to' column "end_time"`

### Metrics (with `metric-sma-for-*` companion datasets)
- **Use for**: Resource utilization, SLI/SLO tracking, capacity planning

### Reference Tables
- Reference tables (e.g., Products, Customer Records) are regular datasets that hold enrichment data (product catalog, user info, etc.)
- **Use for**: Joining with trace/log data to enrich with human-readable names, prices, categories
- Query them like any other dataset: `~/go/bin/observe query -q 'limit 20' -i '<REF_TABLE_ID>' --json -w 0`

## Investigation Workflows

### Error Investigation Playbook

When users report failures, follow this multi-step approach:

1. **Get an overview** — sample recent records to understand schema:
   ```
   ~/go/bin/observe query -q 'limit 5' -i '<DATASET_ID>' -r 1h -x
   ```

2. **Quantify the problem** — count errors by message type:
   ```
   ~/go/bin/observe query -q 'filter body ~ /error|fail|exception/i | statsby count:count(1), group_by(body)' -i '<LOG_DATASET>' -r 24h
   ```

3. **Identify the blast radius** — break down by pod/service/namespace:
   ```
   ~/go/bin/observe query -q 'filter body ~ /error/i | statsby count:count(1), group_by(pod)' -i '<LOG_DATASET>' -r 24h
   ```

4. **Check for time patterns** — trend errors over time:
   ```
   ~/go/bin/observe query -q 'filter body ~ /error/i | timechart 1h, count:count(1)' -i '<LOG_DATASET>' -r 24h
   ```

5. **Inspect full error details** — get stack traces and attributes:
   ```
   ~/go/bin/observe query -q 'filter body ~ /error/i | pick_col timestamp, body, attributes | limit 5' -i '<LOG_DATASET>' -r 24h --json -w 0
   ```

6. **Correlate with traces** — find error spans in the tracing dataset:
   ```
   ~/go/bin/observe query -q 'filter service_name = "<SERVICE>" and error = true | pick_col start_time, end_time, span_name, attributes, trace_id | limit 10' -i '<SPAN_DATASET>' -r 24h -x -w 0
   ```

7. **Trace a single request end-to-end** — follow one trace ID across all services:
   ```
   ~/go/bin/observe query -q 'filter trace_id = "<TRACE_ID>" | pick_col start_time, end_time, service_name, span_name, error, duration, attributes' -i '<SPAN_DATASET>' -r 24h -x -w 0
   ```

8. **Cross-reference dimensions** — break down errors by attributes (e.g., loyalty level, card type):
   ```
   ~/go/bin/observe query -q 'filter service_name = "<SERVICE>" | make_col attr_val: string(attributes["<KEY>"]) | statsby count:count(1), group_by(attr_val, error)' -i '<SPAN_DATASET>' -r 24h
   ```

### Revenue / Business Impact Analysis Playbook

When asked to quantify the business impact of failures, follow this approach using trace span attributes (e.g., `app.payment.amount`):

1. **Get totals by success vs failure** — aggregate revenue and transaction counts:
   ```
   ~/go/bin/observe query -q 'filter service_name = "<SERVICE>" and span_name = "<SPAN>" | make_col amount: float64(attributes["<AMOUNT_KEY>"]) | statsby total: sum(amount), count: count(1), avg_order: avg(amount), min_order: min(amount), max_order: max(amount), group_by(error)' -i '<SPAN_DATASET>' -r 24h -w 0
   ```

2. **Calculate rates** — add hourly burn rate and percentage breakdowns:
   ```opal
   ... | make_col pct_of_total: round(100.0 * count / <TOTAL>.0, 1), hourly_rate: round(total / 24.0, 2)
   ```

3. **Distribution by order size** — bucket failed transactions to find where the biggest dollar losses are:
   ```opal
   filter error = true | make_col amount: float64(attributes["<AMOUNT_KEY>"]) | make_col bucket: case(amount < 100, "< $100", amount < 500, "$100-$500", amount < 1000, "$500-$1K", amount < 5000, "$1K-$5K", amount < 10000, "$5K-$10K", true, "$10K+") | statsby lost: sum(amount), count: count(1), group_by(bucket)
   ```

4. **Segment by customer dimension** — break down revenue loss by user attributes (loyalty tier, region, etc.):
   ```opal
   filter error = true | make_col segment: string(attributes["<SEGMENT_KEY>"]), amount: float64(attributes["<AMOUNT_KEY>"]) | statsby lost: sum(amount), count: count(1), group_by(segment)
   ```

5. **Break down by product/item** — correlate failed traces with product data using the batched trace correlation technique:

   **Step 1**: Export ALL failed trace IDs and split into batches of ~2,000:
   ```bash
   ~/go/bin/observe query -q 'filter service_name = "<SERVICE>" and span_name = "<SPAN>" and error = true | pick_col start_time, end_time, trace_id' -i '<SPAN_DATASET>' -r 24h --json \
     | python3 -c "
   import sys, json
   traces = list(set(json.loads(l)['trace_id'] for l in sys.stdin if l.strip()))
   print(f'Total unique failed trace IDs: {len(traces)}', file=sys.stderr)
   batch_size = 2000
   for i in range(0, len(traces), batch_size):
       open(f'/tmp/failed_batch_{i//batch_size}.txt','w').write('|'.join(traces[i:i+batch_size]))
   print(f'Created {(len(traces)+batch_size-1)//batch_size} batches')
   "
   ```

   **Step 2**: Run all batches in parallel to find correlated spans (e.g., product lookups):
   ```bash
   NUM_BATCHES=<N>  # from step 1
   for i in $(seq 0 $((NUM_BATCHES-1))); do
     TRACES=$(cat /tmp/failed_batch_${i}.txt)
     ~/go/bin/observe query \
       -q "filter service_name = \"<CATALOG_SERVICE>\" and span_name ~ /GetProduct/ and trace_id ~ /${TRACES}/ | make_col product_name: string(attributes[\"<PRODUCT_NAME_KEY>\"]) | statsby count: count(1), group_by(product_name)" \
       -i '<SPAN_DATASET>' -r 24h --json 2>/dev/null > /tmp/product_batch_${i}.json &
   done
   wait
   ```

   **Step 3**: Aggregate all batch results and calculate price-weighted revenue:
   ```python
   import json, glob
   product_counts = {}
   for f in glob.glob('/tmp/product_batch_*.json'):
       for line in open(f):
           obj = json.loads(line.strip())
           name = obj['product_name']
           product_counts[name] = product_counts.get(name, 0) + int(obj['count'])

   # Price-weight using reference table data
   total_weighted = sum(prices[name] * count for name, count in product_counts.items())
   for name, count in product_counts.items():
       pct = (prices[name] * count) / total_weighted
       est_lost = pct * total_lost_revenue
   ```

   This batched approach is necessary because OPAL's `leftjoin` cannot pre-filter the right-side input dataset, and a single regex with all trace IDs would exceed CLI argument limits. Batches of ~2,000 trace IDs each work well within limits.

   **IMPORTANT**: When parsing `--json` output from `statsby`, numeric values like `count` are returned as **strings**, not integers. Always cast with `int()` in Python.

6. **Present the results** — always include:
   - Total lost revenue (absolute dollar amount)
   - Failure rate (% of transactions)
   - Hourly/daily burn rate
   - Projected loss if unresolved (extrapolate to 30 days)
   - Distribution by order size (high-value orders often drive most of the dollar loss)
   - Per-product/item breakdown with unit prices when reference data is available
   - Severity recommendation (P0/P1/P2) based on magnitude

### OPAL Tips for Numeric Analysis
- Use `float64(attributes["key"])` to extract numeric values from JSON attributes for `sum()`, `avg()`, `min()`, `max()`.
- Use `round(value, N)` to control decimal precision in results.
- Use `case()` for bucketing continuous values into discrete ranges.
- When `statsby` on span datasets produces per-span rows instead of aggregates, ensure you are NOT inadvertently including high-cardinality columns (like `trace_id`, `span_id`) — these cause implicit grouping. Use only the `group_by()` columns you intend.

### Cross-Trace Correlation Technique (Batched Parallel Queries)

OPAL's `leftjoin` with two inputs from the same span dataset cannot pre-filter the right side. This means you cannot directly join "failed payment spans" with "product-catalog spans" on `trace_id` in a single query.

**Workaround — Batched Parallel Queries**:
1. Export ALL trace IDs from one query (e.g., failed payment spans) using `--json`
2. Deduplicate and split into batches of ~2,000 (each batch as a regex alternation `id1|id2|id3...`)
3. Run all batches in parallel using `&` and `wait`
4. Aggregate results from all batch output files

```bash
# Step 1: Export all trace IDs and split into batches
~/go/bin/observe query -q '... | pick_col start_time, end_time, trace_id' -i '<DATASET>' -r 24h --json \
  | python3 -c "
import sys, json
traces = list(set(json.loads(l)['trace_id'] for l in sys.stdin if l.strip()))
batch_size = 2000
for i in range(0, len(traces), batch_size):
    open(f'/tmp/batch_{i//batch_size}.txt','w').write('|'.join(traces[i:i+batch_size]))
num = (len(traces)+batch_size-1)//batch_size
open('/tmp/num_batches.txt','w').write(str(num))
print(f'{len(traces)} traces -> {num} batches')
"

# Step 2: Run all batches in parallel
NUM=$(cat /tmp/num_batches.txt)
for i in $(seq 0 $((NUM-1))); do
  TRACES=$(cat /tmp/batch_${i}.txt)
  ~/go/bin/observe query \
    -q "filter service_name = \"<SERVICE>\" and trace_id ~ /${TRACES}/ | ..." \
    -i '<DATASET>' -r 24h --json 2>/dev/null > /tmp/result_${i}.json &
done
wait

# Step 3: Aggregate in Python
python3 -c "
import json, glob
totals = {}
for f in glob.glob('/tmp/result_*.json'):
    for line in open(f):
        obj = json.loads(line.strip())
        key = obj['<GROUP_KEY>']
        totals[key] = totals.get(key, 0) + int(obj['count'])
for k, v in sorted(totals.items(), key=lambda x: -x[1]):
    print(f'{k}: {v}')
"
```

**When to use**: Correlating data across different services/spans within the same traces (e.g., "which products were in orders that had payment failures"). Use this to cover 100% of traces instead of sampling.

**Why batches of ~2,000**: Each trace ID is 32 hex chars. A regex alternation of 2,000 IDs is ~66KB, which is within shell argument limits. Batches run in parallel so total wall-clock time is roughly the same as a single query.

**Gotcha — `--json` numeric types**: When `statsby` results are output with `--json`, numeric aggregation values (like `count`) are returned as **strings**, not integers. Always cast with `int()` or `float()` when parsing in Python.

### Tips for Investigation
- **Start broad, then narrow**: Begin with high-level counts before drilling into specifics.
- **Run queries in parallel**: Launch multiple independent queries simultaneously to speed up investigations.
- **Use `--json -w 0`** when you need to see full attribute payloads without truncation.
- **Use `-x` (extended format)** for records with many columns or long values.
- **Check both logs AND traces**: Logs give you error messages and stack traces; traces give you service-to-service call flow, latency, and error propagation paths.
- **Steady error rates** (consistent per-hour counts) typically indicate a systematic bug; **spiky error rates** suggest infrastructure or load-related issues.

## Listing & Getting Objects

### List objects
```
~/go/bin/observe list <object_type> [substring_filter]
```

Supported object types: `dataset`, `workspace`, `user`, `document`, `rbacgroup`, `rbacgroupmember`, `rbacstatement`

Examples:
```
~/go/bin/observe list dataset
~/go/bin/observe list dataset "Container Logs"
~/go/bin/observe list workspace
```

### Get object details
```
~/go/bin/observe get <object_type> <object_id>
```

Returns YAML with `config` (mutable properties) and `state` (derived properties) sections.

Example:
```
~/go/bin/observe get dataset 41042071
~/go/bin/observe get workspace 41042069
```

## Deleting Objects

```
~/go/bin/observe delete <object_type> <object_id>
```

Example:
```
~/go/bin/observe delete document o::1234567890:document:8007654321
```

## Uploading Documents

Upload documents (e.g., prompts for o11y help tool):

```
~/go/bin/observe upload prompt my-notes.md
~/go/bin/observe upload prompt --as-filename oncall-schedule.md path/to/new/file.md
```

Prompt documents are indexed for the o11y help tool. Up to 500 prompt documents are supported.

## Global Configuration Flags

| Flag | Short | Description |
|------|-------|-------------|
| `--customerid` | `-C` | Numeric tenant ID |
| `--site` | `-S` | Tenant site domain |
| `--authtoken` | `-A` | Bearer token (without "Bearer" prefix) |
| `--profile` | `-P` | Config profile name (default: `default`) |
| `--config` | | Config file path (default: `~/.config/observe.yaml`) |
| `--workspace` | | Default workspace for objects |
| `--output` | `-O` | Output file (default: stdout) |
| `--debug` | `-D` | Extra debug logging |
| `--quiet` | `-Q` | Suppress info logs |
| `--show-config` | | Print resolved config before running |

## Guidelines

- Always use the full binary path `~/go/bin/observe` when running commands.
- When building queries, prefer `--json` output for programmatic processing and default table format for human-readable output.
- Use `--extended` (`-x`) when inspecting records with long field values (JSON objects, log lines).
- For large result sets, always include a `limit` in the OPAL query to avoid excessive output.
- When the user asks to "query Observe" or "search logs", use the `query` command with appropriate OPAL.
- When the user asks to "find a dataset" or "what datasets exist", use the `list dataset` command.
- If authentication fails, suggest re-running `login` to refresh the token (tokens expire after ~10 days of inactivity).

## Gotchas & Common Pitfalls

- **Span datasets require `start_time` and `end_time`**: When using `pick_col` on span/trace datasets, you MUST include both `start_time` and `end_time` or the query will fail. This is because spans are interval-type data.
- **Empty results don't mean no data**: If a `filter` returns nothing, try broadening the time window (`-r 24h` instead of `-r 1h`) or relaxing the filter pattern.
- **`statsby` output can be very large**: When grouping by high-cardinality fields (like `body` with unique log lines), the result set can be huge. Add a `| topk 20, count` or similar to limit.
- **Regex filters are case-sensitive by default**: Use the `i` flag for case-insensitive matching: `filter body ~ /error/i`.
- **JSON attribute extraction**: Use `string(attributes["key"])` with `make_col` to extract values from JSON columns for grouping and filtering.
- **Truncated output**: Default column width is 64 chars. Use `-w 0` to see full values, or `--json` for complete attribute payloads.
- **Large result files**: When output exceeds ~30KB, it gets saved to a file. Use `--json` with `limit` to keep output manageable.