--- name: configuring-opentelemetry-dotnet description: Configure OpenTelemetry distributed tracing, metrics, and logging in ASP.NET Core using the .NET OpenTelemetry SDK. Use when adding observability, setting up OTLP exporters, creating custom metrics/spans, or troubleshooting distributed trace correlation. license: MIT --- # Configuring OpenTelemetry in .NET ## When to Use - Adding distributed tracing to an ASP.NET Core application - Setting up OpenTelemetry exporters (OTLP is the primary protocol; Jaeger accepts OTLP natively; Prometheus OTLP ingestion requires explicit opt-in) - Creating custom metrics or trace spans for business operations - Troubleshooting distributed trace context propagation across services ## When Not to Use - The user wants application-level logging only (use ILogger, Serilog) - The user is using Application Insights SDK directly (different API) - The user needs APM with a commercial vendor's proprietary SDK ## Inputs | Input | Required | Description | |-------|----------|-------------| | ASP.NET Core project | Yes | The application to instrument | | Observability backend | No | Where to export: OTLP collector, Aspire dashboard, Jaeger (accepts OTLP natively) | ## Workflow ### Step 1: Install the correct packages **There are many OpenTelemetry NuGet packages. Install exactly these:** ```bash # Core SDK + ASP.NET Core instrumentation + logging integration dotnet add package OpenTelemetry.Extensions.Hosting dotnet add package OpenTelemetry.Instrumentation.AspNetCore dotnet add package OpenTelemetry.Instrumentation.Http # Exporter dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol # OTLP exporter for traces, metrics, AND logs # Optional — dev/local debugging only (do NOT include in production deployments) # dotnet add package OpenTelemetry.Exporter.Console ``` **Do NOT install `OpenTelemetry` alone** — you need `OpenTelemetry.Extensions.Hosting` for proper DI integration. #### Optional: additional auto-instrumentation packages Install only the packages that match the libraries your application uses: ```bash dotnet add package OpenTelemetry.Instrumentation.SqlClient # SQL Server queries dotnet add package OpenTelemetry.Instrumentation.EntityFrameworkCore # EF Core dotnet add package OpenTelemetry.Instrumentation.GrpcNetClient # gRPC calls dotnet add package OpenTelemetry.Instrumentation.Runtime # GC, thread pool metrics ``` ### Step 2: Configure all signals in Program.cs ```csharp using OpenTelemetry.Resources; using OpenTelemetry.Trace; using OpenTelemetry.Metrics; using OpenTelemetry.Logs; var builder = WebApplication.CreateBuilder(args); builder.Services.AddOpenTelemetry() .ConfigureResource(resource => resource .AddService(serviceName: builder.Environment.ApplicationName)) .WithTracing(tracing => tracing .AddAspNetCoreInstrumentation(options => { // Filter out health check endpoints from traces options.Filter = httpContext => !httpContext.Request.Path.StartsWithSegments("/healthz"); }) .AddHttpClientInstrumentation(options => { options.RecordException = true; }) // Optional: add SQL instrumentation if using SqlClient directly // .AddSqlClientInstrumentation(options => // { // options.SetDbStatementForText = true; // options.RecordException = true; // }) // Custom activity sources (must match ActivitySource names in your code) .AddSource("MyApp.Orders") .AddSource("MyApp.Payments") .AddSource("MyApp.Messaging")) .WithMetrics(metrics => metrics .AddAspNetCoreInstrumentation() .AddHttpClientInstrumentation() // Optional: .AddRuntimeInstrumentation() for GC and thread pool metrics // (requires OpenTelemetry.Instrumentation.Runtime package) // Custom meters (must match Meter names in your code) .AddMeter("MyApp.Metrics")) .WithLogging(logging => { logging.IncludeScopes = true; // logging.IncludeFormattedMessage = true; // Enable if you need the formatted message string in log exports }) // Single OTLP exporter for all signals — reads OTEL_EXPORTER_OTLP_ENDPOINT // env var (defaults to http://localhost:4317). Override via environment variable // or appsettings.json configuration. .UseOtlpExporter(); ``` ### Step 3: Understanding log–trace correlation The `.WithLogging()` call in Step 2 integrates ILogger with OpenTelemetry: - Each log entry automatically includes TraceId and SpanId for correlation with traces - The service resource from `.ConfigureResource()` propagates to logs automatically - `UseOtlpExporter()` applies to logs alongside traces and metrics - No additional packages or separate `SetResourceBuilder` call needed ### Step 4: Create custom spans (Activities) for business operations ```csharp using System.Diagnostics; using Microsoft.Extensions.Logging; public class OrderService { // Create an ActivitySource matching what you registered in Step 2 private static readonly ActivitySource ActivitySource = new("MyApp.Orders"); private readonly ILogger _logger; public OrderService(ILogger logger) => _logger = logger; public async Task ProcessOrderAsync(CreateOrderRequest request) { // Start a new span using var activity = ActivitySource.StartActivity("ProcessOrder"); // Add attributes (tags) to the span activity?.SetTag("order.customer_id", request.CustomerId); activity?.SetTag("order.item_count", request.Items.Count); try { // Child span for validation using (var validationActivity = ActivitySource.StartActivity("ValidateOrder")) { await ValidateOrderAsync(request); validationActivity?.SetTag("validation.result", "passed"); } // Child span for payment using (var paymentActivity = ActivitySource.StartActivity("ProcessPayment", ActivityKind.Client)) // Client = outgoing call { paymentActivity?.SetTag("payment.method", request.PaymentMethod); await ProcessPaymentAsync(request); } var order = new Order { Id = Guid.NewGuid(), CustomerId = request.CustomerId, Status = "Completed" }; activity?.SetTag("order.status", "completed"); activity?.SetStatus(ActivityStatusCode.Ok); return order; } catch (Exception ex) { activity?.SetStatus(ActivityStatusCode.Error, ex.Message); // Log via ILogger — OpenTelemetry captures this with trace correlation. // Prefer logging over activity.RecordException() as OTel is deprecating // span events for exception recording in favor of log-based exceptions. _logger.LogError(ex, "Order processing failed for customer {CustomerId}", request.CustomerId); throw; } } } ``` **Critical: `ActivitySource` name must match `AddSource("...")` in configuration.** Unmatched sources are silently ignored — this is the #1 debugging issue. ### Step 5: Create custom metrics Use `IMeterFactory` (injected via DI) to create meters — this ensures proper lifetime management and testability. ```csharp using System.Diagnostics; using System.Diagnostics.Metrics; public class OrderMetrics { private readonly Counter _ordersProcessed; private readonly Histogram _orderProcessingDuration; private readonly UpDownCounter _activeOrders; public OrderMetrics(IMeterFactory meterFactory) { // Meter name must match AddMeter("...") in configuration var meter = meterFactory.Create("MyApp.Metrics"); // Counter — use for things that only go up _ordersProcessed = meter.CreateCounter( "orders.processed", "orders", "Total orders successfully processed"); // Histogram — use for measuring distributions (latency, sizes) _orderProcessingDuration = meter.CreateHistogram( "orders.processing_duration", "ms", "Time to process an order"); // UpDownCounter — use for things that go up AND down _activeOrders = meter.CreateUpDownCounter( "orders.active", "orders", "Currently processing orders"); } public void RecordOrderProcessed(string region, double durationMs) { // Tags enable dimensional filtering (by region, status, etc.) var tags = new TagList { { "region", region }, { "order.type", "standard" } }; _ordersProcessed.Add(1, tags); _orderProcessingDuration.Record(durationMs, tags); } } ``` Register `OrderMetrics` in DI: ```csharp builder.Services.AddSingleton(); ``` ### Step 6: Configure context propagation for distributed scenarios Trace context propagation is automatic for HTTP calls when using `AddHttpClientInstrumentation()`. For non-HTTP scenarios: ```csharp using System; using System.Collections.Generic; using System.Diagnostics; using OpenTelemetry.Context.Propagation; // ActivitySource should be static — register via .AddSource("MyApp.Messaging") in Step 2 private static readonly ActivitySource MessageSource = new("MyApp.Messaging"); // Manual context propagation (e.g., across message queues) // On the SENDING side: var propagator = Propagators.DefaultTextMapPropagator; var activityContext = Activity.Current?.Context ?? default; var context = new PropagationContext(activityContext, Baggage.Current); var carrier = new Dictionary(); propagator.Inject(context, carrier, (dict, key, value) => dict[key] = value); // Send carrier dictionary as message headers // On the RECEIVING side: var parentContext = propagator.Extract(default, carrier, (dict, key) => dict.TryGetValue(key, out var value) ? new[] { value } : Array.Empty()); Baggage.Current = parentContext.Baggage; using var activity = MessageSource.StartActivity("ProcessMessage", ActivityKind.Consumer, parentContext.ActivityContext); // Links to parent trace! ``` ## Validation - [ ] Traces appear in the observability backend (Jaeger, Aspire dashboard, etc.) - [ ] HTTP requests automatically create spans with correct verb, URL, status code - [ ] Custom `ActivitySource` names match `AddSource()` registrations - [ ] Custom `Meter` names match `AddMeter()` registrations - [ ] Logs include TraceId and SpanId for correlation - [ ] Health check endpoints are filtered from traces - [ ] Exception details appear on error spans ## Common Pitfalls | Pitfall | Solution | |---------|----------| | `ActivitySource.StartActivity` returns null | Source name doesn't match any `AddSource()` — names must match exactly | | Traces not appearing in exporter | Check OTLP endpoint: gRPC uses port 4317, HTTP uses 4318 | | Missing HTTP client spans | Ensure `AddHttpClientInstrumentation()` is registered; it works for both `IHttpClientFactory`/DI and `new HttpClient()` (use `IHttpClientFactory` for lifetime management) | | High cardinality tags | Don't use user IDs, request IDs, or UUIDs as metric tags — explodes storage | | OTLP gRPC vs HTTP mismatch | Default is gRPC (port 4317); if collector only accepts HTTP, set `OtlpExportProtocol.HttpProtobuf` | | `Meter` / `ActivitySource` lifecycle | `ActivitySource` should be static; create `Meter` via `IMeterFactory` from DI (not `new Meter()`) for proper lifetime management and testability |