# Distributed Tracing: X-Ray and ADOT

X-Ray SDK is in maintenance mode. Use ADOT (OpenTelemetry) for all new projects.

## Contents

- [ADOT vs X-Ray SDK](#adot-vs-x-ray-sdk)
- [Trace structure](#trace-structure)
- [Annotations vs metadata](#annotations-vs-metadata)
- [Sampling rules](#sampling-rules)
- [ADOT collector configuration](#adot-collector-configuration)
- [Instrumentation patterns](#instrumentation-patterns)
- [Migration constraints](#migration-constraints-x-ray-sdk--otel)
- [Common mistakes](#common-mistakes)

---

## ADOT vs X-Ray SDK

| Criteria | X-Ray SDK | ADOT (OpenTelemetry) |
|----------|----------|---------------------|
| Status | **Maintenance mode** | Actively developed |
| Multi-backend | X-Ray only | CloudWatch, X-Ray, Prometheus, OpenSearch |
| Auto-instrumentation | Limited | Java, Python (compute); Node.js (Lambda layer only) |
| Vendor lock-in | AWS-specific | Vendor-neutral (OTel standard) |
| Lambda support | Built-in daemon | Lambda layer (auto-instrumentation) |
| **Recommendation** | **Legacy apps only** | **All new projects** |

**Migration path**: AWS provides migration guides from X-Ray SDK to OpenTelemetry SDK. The CloudWatch agent now also supports sending traces to X-Ray — no separate daemon needed.

---

## Trace structure

- **Trace** — collection of all segments from a single request, identified by trace ID
- **Segment** — JSON document with a **64 KB** documented limit representing work done by a service. Do not exceed this; behavior above 64 KB is undocumented and may change.
- **Subsegment** — granular detail within a segment (downstream calls, custom code blocks)
- **Inferred segment** — generated by X-Ray from subsegments for uninstrumented downstream services

### Trace ID format

```
X-Amzn-Trace-Id: Root=1-58406520-a006649127e371903a2de979;Parent=53995c3f42cd8ad8;Sampled=1
```

Format: `1-{8 hex epoch}-{24 hex unique}`. W3C trace IDs are supported (reformatted).

### Retention

- Trace data: **30 days** (not configurable)
- Service graph: **30 days**

---

## Annotations vs metadata

| Feature | Annotations | Metadata |
|---------|------------|----------|
| **Indexed** | Yes — Searchable with filter expressions | No — Not indexed |
| **Value types** | String, Number, Boolean only | Any type (objects, arrays) |
| **Limit** | **50 indexed per trace** (API accepts more, but only 50 are searchable) | No limit (within segment size) |
| **Key format** | Alphanumeric + underscore only | Any key (`AWS.` prefix reserved) |
| **Use case** | Filtering/grouping traces | Storing debug data |

**Rule of thumb**: If you need to search for it → annotation. If you just need to store it → metadata.

**WARNING**: 50 annotations per trace is a hard limit. Plan your annotation schema carefully.

---

## Sampling rules

### Default rule

- **Reservoir**: 1 request per second (shared across all instances)
- **Rate**: 5% of additional requests
- Conservative default to control costs

### Rule evaluation

- Rules evaluated in ascending **priority** order (1–9999, lower = higher priority)
- Default rule priority = 10000 (always last)
- First matching rule wins

### Rule parameters

| Parameter | Description |
|-----------|-------------|
| Priority | 1–9999 (lower = higher priority) |
| Reservoir | Fixed traces/second before applying rate |
| Rate | Percentage of additional requests (0–100 in console, 0.0–1.0 in API/JSON) |
| Service name | Wildcards `*` and `?` supported |
| Service type | e.g., `AWS::EC2::Instance`, `AWS::Lambda::Function` |
| HTTP method | GET, POST, etc. |
| URL path | Path portion of URL |

### Parent-based sampling (critical concept)
Sampling decision is made **once** by the root service. Downstream services honor the upstream decision regardless of their own rules. Custom rules only apply where no sampling decision exists yet.

### Adaptive sampling (newer)

- `SamplingRateBoost` — auto-increases rate during anomalies
- `MaxRate` — ceiling for boosted rate
- `CooldownWindowMinutes` — prevents continuous boosts (recommended when SamplingRateBoost is configured)

---

## ADOT collector configuration

### Architecture

```
[Receivers] → [Processors] → [Exporters]
```

### CloudWatch + X-Ray pipeline

> The `0.0.0.0` receivers below listen on every interface with no TLS and no authentication. Bind
> them to `127.0.0.1` when the senders are on the same host; otherwise it is recommended to restrict
> the OTLP `4317` / `4318` ports via security groups or host firewall and to avoid co-locating
> untrusted workloads. This guide does not apply those controls, so assess and configure them for
> your environment.

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 30s
    send_batch_size: 8192

exporters:
  awsxray:
    region: us-east-1
  awsemf:
    namespace: MyApplication
    region: us-east-1

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsxray]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsemf]
```

### EKS DaemonSet deployment

```yaml
resources:
  limits:
    memory: 200Mi
  requests:
    cpu: 250m
    memory: 100Mi
```

### Cardinality prevention (three-layer defense)

1. **OTel SDK level**: Don't emit high-cardinality attributes (ContainerID, CustomerID, RequestID)
2. **ADOT Collector level**: Use Filter Processor to drop metrics by name/attribute
3. **Backend level**: Use backend-specific dimension filtering (CloudWatch: `dimension_rollup_option` + `metric_declarations`; Prometheus: `metric_relabel_configs`)

Filter as early as possible in the pipeline to reduce cost and cardinality.

---

## Instrumentation patterns

### Lambda: enable active tracing (CDK)

```typescript
import { Tracing } from 'aws-cdk-lib/aws-lambda';

const fn = new lambda.Function(this, 'MyFunction', {
  runtime: lambda.Runtime.NODEJS_20_X,
  handler: 'index.handler',
  code: lambda.Code.fromAsset('lambda'),
  tracing: Tracing.ACTIVE,
});
```

### API Gateway: enable tracing

```typescript
const api = new apigateway.RestApi(this, 'MyApi', {
  deployOptions: {
    tracingEnabled: true,
  },
});
```

Or via CLI: `aws apigateway update-stage --rest-api-id <id> --stage-name prod --patch-operations op=replace,path=/tracingEnabled,value=true`

### Trace-log correlation
Inject trace ID into application logs for cross-pillar correlation:

```python
import logging
from opentelemetry import trace

ctx = trace.get_current_span().get_span_context()
trace_id = format(ctx.trace_id, '032x')
logging.info("Processing request", extra={"trace_id": trace_id})
```

---

## Migration constraints (X-Ray SDK → OTel)

### Annotations require explicit opt-in
In OTel, all span attributes become X-Ray **metadata** by default. To make an attribute a searchable X-Ray annotation, add its key to the `aws.xray.annotations` list:

```python
span.set_attribute("aws.xray.annotations", ["order_id", "customer_tier"])
span.set_attribute("order_id", "12345")
```

Without this, you lose all annotation-based filtering after migration.

### Centralized sampling requires a proxy
The ADOT collector config must include the `awsproxy` extension (or use the CloudWatch agent as a proxy) for X-Ray centralized sampling rules to work. Without a proxy, the SDK falls back to a default local rule (1 req/sec + 5%):

```yaml
extensions:
  awsproxy:
    endpoint: 127.0.0.1:2000
service:
  extensions: [awsproxy]
```

SDK env vars: `OTEL_TRACES_SAMPLER=xray` and `OTEL_TRACES_SAMPLER_ARG=endpoint=http://localhost:2000`

Centralized sampling language support: Java, .NET, Python, Node.js (ADOT). Vanilla OTel SDK: Java, .NET, Go.

### Mixed propagation during incremental migration
OTel defaults to W3C Trace Context; X-Ray SDK uses X-Ray trace header. During migration, configure both:

```
OTEL_PROPAGATORS=xray,tracecontext
```

Without this, traces break at service boundaries between old and new instrumentation.

### Port conflict: stop X-Ray daemon before starting ADOT
Both use port 2000. Running both simultaneously causes silent data loss.

### Lambda ADOT layer adds cold start latency
ADOT Lambda layers increase memory usage and cold start time. For latency-sensitive functions where you don't need OTel's multi-backend capabilities, X-Ray SDK may still be preferable.

### W3C trace ID version requirement
ADOT Collector 0.34.0+ (X-Ray Exporter 0.86.0+) is required to accept W3C-format trace IDs. Older versions silently reject them.

---

## Common mistakes

1. **Using X-Ray SDK for new projects** — Maintenance mode. Use ADOT/OpenTelemetry.

2. **Storing searchable data as metadata** — Metadata is NOT indexed. Use annotations for data you need to filter by.

3. **Exceeding 50 annotations per trace** — Hard limit. Plan your annotation schema.

4. **Not stripping X-Amzn-Trace-Id from untrusted requests** — Users can inject trace IDs or sampling decisions.

5. **Default sampling for all services** — 1 req/sec + 5% is too conservative for low-traffic services (may miss issues) and too aggressive for high-traffic (unnecessary cost). Tune per service.

6. **StepFunctions tracing overrides Lambda** — When StepFunction tracing is enabled, downstream Lambda tracing is always enabled regardless of Lambda's own config.

7. **Cross-account tracing** — Trace IDs propagate naturally across accounts, but unified cross-account viewing requires CloudWatch Observability Access Manager (OAM) setup with monitoring/source account links.