Guido van Tricht

OpenTelemetry - .NET Setup

During the past couple of weeks, I have been working on optimizing the monitoring and tracing setup for one of my projects. We were already using Prometheus and Grafana, but didn’t really take advantage of the full feature set.

In an effort to build in more tracing capabilities and gather more data for easier troubleshooting of issues, I setup OpenTelemetry for the .NET (Core) applications part of the platform.

In this article I’ll describe how to install the OpenTelemetry SDK in .NET and how to extend your code to expose more data in the metrics and tracing output.

Install OpenTelemetry SDK

The .NET OpenTelemetry SDK consists of multiple NuGet packages that enable different features.
To install the base setup, you will want to install the following packages

  • OpenTelemetry.Extensions.Hosting → Base package for exposing a /metrics endpoint
  • OpenTelemetry.Instrumentation.AspNetCore → Base package to expose Asp.NET Core metrics in the /metrics endpoint
  • OpenTelemetry.Exporter.Console → Base package to log metrics and tracing to the console output

Other packages that may be of interest are

  • OpenTelemetry.Exporter.OpenTelemetryProtocol → Package enabling tracing exporting to OpenTelemetry capable tooling, such as .NET Aspire dashboard and Jaeger
  • OpenTelemetry.Instrumentation.Http → Package to enable HttpClient metrics to be output in metrics and tracing
  • OpenTelemetry.Instrumentation.Process → Package to enable process metrics to be output from the /metrics endpoint
  • OpenTelemetry.Instrumentation.Runtime → Package to enable runtime metrics to be output from the /metrics endpoint
  • OpenTelemetry.Instrumentation.StackExchangeRedis → Package to enable Redis session storage metrics to be output in metrics and tracing

The versions of these package you need to install is based on the .NET version you use. Note that many OpenTelemetry packages are marked as prerelease version.

  • For .NET 6, use package version <= 1.9.0
  • For .NET 8+, use package versions > 1.11.0

Enable OpenTelemetry

To enable the OpenTelemetry SDK, we first need to register it with DependencyInjection in the IServiceCollection.

// Register OpenTelemetry
var otel = services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        // Specify a unique name for you application type
        // this will be used to separate tracing logs
        .AddService("[Application name]"));
 
// Enable Metrics
otel.WithMetrics(metrics => metrics
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    // Enable any additional Instrumentations
    // .AddRuntimeInstrumentation()
    // .AddProcessInstrumentation()
    // Enable the Prometheus Exporter
    .AddPrometheusExporter());
 
// Enable Tracing Exporter
otel.WithTracing(tracing =>
{
    // Provide a unique application name which all the activities will be tagged with
    tracing.AddSource("[Application name]");
    tracing.AddAspNetCoreInstrumentation();
    tracing.AddHttpClientInstrumentation();
    // Add any additional Instrumenations
    // tracing.AddRedisInstrumentation();
    // Specify where to export the traces to, like a Jaeger endpoint
    tracing.AddOtlpExporter(otlpOptions =>
    {
        otlpOptions.Endpoint = new Uri("[Endpoint URL]");
        otlpOptions.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.HttpProtobuf;
    });
});

With these services registered, all metrics will be gathered, and traces will be sent to the configured endpoint.

The last thing to do is to enable the /metrics endpoint, by add the Middleware in the IApplicationBuilder:

app.UseOpenTelemetryPrometheusScrapingEndpoint();

Tracing improvements

Out of the box, tracing will be exported with a default set of tags. Depending on your application, you may want to customize this setup.

Filter requests

You may not want to export all requests to you application as traces, such as the /metrics or healthcheck requests.
To exclude these requests, you can configure filters in the tracing.AddAspNetCoreInstrumentation step:

tracing.AddAspNetCoreInstrumentation(options =>
{
    options.Filter = (httpContext) =>
    {
        // Specify paths that you want to exclude
        string[] filters = new string[] {
            "/liveness",
            "/hc",
            "/metrics",
            "/js",
            "/img",
        };
 
        try
        {
            if (filters.Any(f => httpContext.Request.Path.Value.StartsWith(f)))
            {
                return false;
            }
            return true;
        }
        catch
        {
            return true;
        }
    };
});

Custom activity DisplayName and tags

The OpenTelemetry SDK will try to determine the routing tags and the name of an activity by using the HttpRequest and HttpResponse objects and the Route that was triggered during the request.
When using wildcard Routes for example, you will want to customize these values so you can separate traces from each other.

To do so, you can use the EnrichWithHttpRequest and EnrichWithHttpResponse options in the tracing.AddAspNetCoreInstrumentation step:

tracing.AddAspNetCoreInstrumentation(options =>
{
    options.EnrichWithHttpRequest = (activity, httpRequest) =>
    {
        // Customize tag values
        activity.SetTag("http.route", httpRequest.Path.Value);
    };
    options.EnrichWithHttpResponse = (activity, httpResponse) =>
    {
        // Customize the Sitecore .NET SDK route Display Name
        if (activity.DisplayName.Contains("**sitecoreRoute"))
            activity.DisplayName = $"{httpResponse.HttpContext.Request.Method} {httpResponse.HttpContext.Request.Path}";
    };
});

Adding Activities

You can break down traces into multiple smaller activities by using the Activity framework that is built into .NET.

Activities are started from an ActivitySource. Since creating new instances of the ActivitySource object can be expensive, it is best to create one as a Singleton in your application.
Take below InstrumentationSource for example which can be added to the IServiceCollection as a Singleton.

public sealed class InstrumentationSource : IDisposable
{
    internal const string ActivitySourceName = "[Application name]";
 
    public InstrumentationSource()
    {
        string? version = typeof(InstrumentationSource).Assembly.GetName().Version?.ToString();
        this.ActivitySource = new ActivitySource(ActivitySourceName, version);
    }
 
    public ActivitySource ActivitySource { get; }
 
    public void Dispose()
    {
        this.ActivitySource.Dispose();
    }
}

Whenever you want to start a new activity, you can use the InstrumentationSource.ActivitySource like below.

using Activity activity = _instrumentationSource.ActivitySource.StartActivity("[Activity name]");
 
... perform activities here

As Activity is Disposable, you will want to either create an instance in a using or dispose of it at the end of your method.

You will want to create Activities for expensive actions that take place during a thread.