Observability

Scout supports observability. The goal is to establish OpenTelemetry as the Observability framework.

Current State

Signal Support Tested OpenTelemetry Support

Traces

supported, based on OpenTelemetry disabled by default (see configuration property scout.otel.tracing.enabled and scout.otel.initializerEnabled)

(✅)
currently being tested

Metrics

supported, based on OpenTelemetry
disabled by default (see configuration property scout.otel.initializerEnabled)

✅

Logs

supported, based on SLF4J

❌
no integration in OpenTelemetry yet

How Does It Work?

Scout uses the features of OpenTelemetry SDK Autoconfigure, which allows environment-based configuration of the OpenTelemetry SDK.

The main focus is on manual instrumentation provided by the Scout framework. The OpenTelemetry SDK is initialized by the class org.eclipse.scout.rt.opentelemetry.sdk.OpenTelemetryInitializer. This initializer will set up the io.opentelemetry.api.GlobalOpenTelemetry instance and some defaults:

  • Exporters: otlp [1] for metrics and traces (none for logs due to missing (tested) support)
    Protocol: http/protobuf

  • Logical service name: Scout’s ApplicationName

  • Service resource attributes: e.g. service.instance.id (=Scout’s NodeIdentifier)

All these defaults can be changed by the corresponding system property or environment variable, see the README of OpenTelemetry SDK Autoconfigure.

Traces: Context Propagation

It is important to properly propagate the OpenTelemetry Context between threads in order to correctly relate spans and preserve the hierarchy of the calls. Context propagation is especially relevant in Scout Applications when calls are made between Scout UI and Scout backend server and within the Scout UI server when we go from Scout UI Thread to the Scout Model Thread.

Overview of the Context propagation:

Class

Notes

OpenTelemetryFilter

The parent Context is extracted from the incoming HttpRequest (header). Relevant for both UI and backend to capture incoming calls.

HttpServiceTunnel

The Context is injected into a header of the outgoing HttpRequest. This is a relevant instrumentation to capture calls from UI to backend.

ScoutOpenTelemetryContextStorage

Everytime a new Context is made current, that Context is stored on the RunContext. This ensures proper propagation between UI thread and Model thread.

OpenTelemetryContextProcessor

Ensures that the correct Context is propagated between RunContexts.

Setup OpenTelemetry in a Scout Application

Add the SDK related maven dependencies to the maven module(s), e.g. the .server and .ui.html, according to Listing 1.

Listing 1. The dependencies required in the .server and/or .ui.html pom.xml to use the Scout’s OpenTelemetry integration
    <!-- OpenTelemetry -->
    <dependency>
      <groupId>org.eclipse.scout.rt</groupId>
      <artifactId>org.eclipse.scout.rt.opentelemetry.sdk</artifactId>
    </dependency>
    <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-exporter-otlp</artifactId>
    </dependency>

The OpenTelemetry’s Automatic Instrumentation can be used by disabling the Scout’s OpenTelemetryInitializer. by setting the configuration property scout.otel.initializerEnabled to false. Then you can follow the official setup guide.

Tracing

Activate tracing by setting the configuration properties scout.otel.initializerEnabled and scout.otel.tracing.enabled to true. Make sure to include the OpenTelemetryFilter in your setup (e.g by adding it as a IServletFilterContributor or including it in the web.xml). OpenTelemetryFilter should be placed first.

Running It All Together

Under this link you can find a complete docker compose setup of a demo observability infrastructure. Follow these steps to run the infrastructure and prepare the application for observability.

observability sample setup.drawio

  1. Start up the observability infrastructure (for demonstration purposes).
    docker compose up
    To access Grafana go to http://localhost:3000/
    (Prometheus: http://localhost:9090/)

  2. Activate OpenTelemetry SDK & OTLP exporter in the application.

    • Configuration property scout.otel.initializerEnabled=true

    • Maven dependencies, see Listing 1

  3. Use OTLP exporter in dev mode.
    Set configuration property: scout.otel.defaultExporter=otlp or use the system property/environment variables of the autoconfigure feature.

  4. Activate OpenTelemetry Tracing support in the application

    • Configuration property scout.otel.tracing.enabled=true

Use the exporter logging-otlp to just print out the observability data. But do not forget to add the required maven dependency io.opentelemetry:opentelemetry-exporter-logging-otlp.

Providing Custom Metrics

Custom application metrics can be provided by either put the metric handling directly in the production code:

OpenTelemetry openTelemetry = GlobalOpenTelemetry.get();
// continue with https://opentelemetry.io/docs/instrumentation/java/manual/#metrics.

or by providing an implementation of the interface org.eclipse.scout.rt.platform.opentelemetry.IMetricProvider.

public class MyMetricProvider implements IMetricProvider {

  @Override
  public void register(OpenTelemetry openTelemetry) {
    // continue with https://opentelemetry.io/docs/instrumentation/java/manual/#metrics.
  }

  @Override
  public void close() {
    // noop
  }
}

A metric provider is usually used for more generally or feature-independent metrics such as JVM/cpu metrics. It can also be used for metrics whose source code is not under your control, e.g. external libraries.

It is possible to define specific explicit bucket distribution for histogram metrics, see org.eclipse.scout.rt.platform.opentelemetry.IHistogramViewHintProvider

Providing Custom Samplers

The ISamplerCustomizerProvider interface allows you to define custom sampling logic based on span attributes. This can help reduce the number of traces sent to the collector by dropping traces that match specific rules.

In the example below, two sampling rules are defined. If either rule matches (logical OR), the corresponding trace will be dropped. The first rule contains two conditions that must both be met (logical AND).

public class MySamplerProvider implements ISamplerCustomizerProvider {

  @Override
  public Sampler createSamplerCustomizer(Sampler fallback, ConfigProperties config) {
    return new RuleBasedSamplerBuilder(SpanKind.SERVER, fallback)
        .drop(
            EqualityCondition.of(UrlAttributes.URL_QUERY, "poll"),        (1)
            RegexCondition.of(UrlAttributes.URL_PATH, "^/json")           (2)
        )
        .drop(
            RegexCondition.of(UrlAttributes.URL_PATH, "^/status")         (3)
        )
        .build();
  }
}
1 EqualityCondition: The span attribute value is equals to a target value - in this case "poll"
2 RegexCondition: The span attribute value matches a given pattern - in this case "^/json"
3 A single condition forming the second rule, matching the pattern ^/status.

Manual Instrumentation for Tracing

To instrument your own Scout Application see examples for manual instrumentation here

1. OpenTelemetry Protocol - for more information see https://opentelemetry.io/docs/specs/otlp/