Distributed tracing with OpenCensus and Java

Introduction

I tried to implement distributed tracing with OpenCensus API and Java. OpenCensus is a specification based on Google's Cloud Trace (formerly StackDriver Trace) and is a specification for distributed tracing and metric collection. By using this, the PG side can link the backend to Jeager, OpenZipkin, Cloud Trace, etc. while using the unified API.

In fact, OpenCensus is already old and is now standardized as Open Telemetry by integrating with Open Tracing with similar specifications. This includes commercial APMs such as Datadog, NewRelic, Dynatrace, and Instana, so you should use this in the future. So, at first I was playing with Open Telemetry, but although I was able to cooperate with Jeager, it was Cloud Trace does not yet support Java version. For the time being, I tried OpenCensus, which seems to have a lot of documentation.

However, the documents were not organized as much as I expected, so I will summarize them as a reminder.

The code I created this time is below. https://github.com/koduki/miniban/tree/example/opensensus

System configuration

As a system configuration, it simply receives a request with ʻapi-endpoint and throws the process to the backend ʻapi-core with business logic with REST / JSON.

It's not as subdivided as microservices, but it's a common configuration and I think it's good enough to verify distributed tracing. Each API is implemented in JAX-RS using Quarkus. Basically, we don't use MicroProfile-specific API, so any Java EE environment should work as it is.

Implementation

Dependent libraries

Add the following to pom.xml as a dependent library.

<dependency>
    <groupId>io.opencensus</groupId>
    <artifactId>opencensus-api</artifactId>
    <version>0.26.0</version>
</dependency>
<dependency>
    <groupId>io.opencensus</groupId>
    <artifactId>opencensus-impl</artifactId>
    <version>0.26.0</version>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>io.opencensus</groupId>
    <artifactId>opencensus-exporter-trace-stackdriver</artifactId>
    <version>0.26.0</version>
</dependency>
<dependency>
    <groupId>io.opencensus</groupId>
    <artifactId>opencensus-exporter-trace-jaeger</artifactId>
    <version>0.26.0</version>
</dependency>
<dependency>
    <groupId>io.opencensus</groupId>
    <artifactId>opencensus-contrib-http-jaxrs</artifactId>
    <version>0.26.0</version>
</dependency>

The APIs you need are opencensus-api and opencensus-impl. opencensus-exporter-xxx is a variety of exporter libraries that send trace information, and ʻopencensus-contrib-http-jaxrs` is a convenient library for JAX-RS. This time, I have specified two exporters, stackdriver and jaeger, but usually one of them.

Exporter initialization

First, initialize and register the Exporter. This only needs to be done once, so use @Initialized (ApplicationScoped.class) to load it at startup.

Bootstrap.java

@ApplicationScoped
public class Bootstrap {
    public void handle(@Observes @Initialized(ApplicationScoped.class) Object event) {
        JaegerTraceExporter.createAndRegister(
                JaegerExporterConfiguration.builder()
                        .setThriftEndpoint("http://localhost:14268/api/traces")
                        .setServiceName("api-endpoint")
                        .build());
    }
}

Quick Start takes the URL directly as an argument to createAndRegister, but this is already non-existent. It looks like the recommended code, so use JaegerExporterConfiguration.

As you can see, use setThriftEndpoint to specify the URL of the link destination, and use setServiceName to use the service name on Jaeger.

As long as you change the registered part of this Exporter, you can switch the backend arbitrarily. For example, if you want to use Cloud Trace, change as follows.

Bootstrap.java

@ApplicationScoped
public class Bootstrap {
    public void handle(@Observes @Initialized(ApplicationScoped.class) Object event) {
        String gcpProjectId = "your GCP project ID";
        StackdriverTraceExporter.createAndRegister(
                StackdriverTraceConfiguration.builder()
                        .setProjectId(gcpProjectId)
                        .build());
    }
}

Creating a Trace Span

Next is the creation of Span.

try (Scope ss = Tracing.getTracer()
        .spanBuilder("Span Name")
        .setRecordEvents(true)
        .setSampler(Samplers.alwaysSample())
        .startScopedSpan()) {

    // do somthing.
}

Tracing.getTracer () gets the tracer from a singleton or global. From there, use spanBuilder to assemble the Span. The point is setSampler. The sampler specifies how often to get the trace. If you do not specify Samplers.alwaysSample () here, the trace information will not always be written. In production, there may be cases where an appropriate threshold is set in terms of cost and load, but at the time of testing, there are too few requests other than load testing, so be sure to specify always. Details of other samplers can be found at here.

By the way, it is troublesome to write the above every time, so I defined the following helper somed.

public static <R> R trace(Supplier<R> callback) {
    var depth = 2;
    var className = Thread.currentThread().getStackTrace()[depth].getClassName();
    var methodName = Thread.currentThread().getStackTrace()[depth].getMethodName();

    try (var ss = Tracing.getTracer()
            .spanBuilder(className + "$" + methodName)
            .setRecordEvents(true)
            .setSampler(Samplers.alwaysSample())
            .startScopedSpan()) {
        return callback.get();
    }
}

The class name and method name are acquired and automatically specified in the Span name. When using it, it looks like the following.

@GET
@Path("/{userId}/balance")
public Map<String, Long> getBalance(@PathParam("userId") String userId) {
    return trace(() -> {
        var balance = service.getBalance(userId);
        return balance;
    });
}

Propagation of Trace Context

If you are in the same application, just do startScopedSpan as above and the nested Span will be created without permission. However, the heart of distributed tracing is inter-system cooperation. Therefore, it is necessary to link remote contexts. However, from this point on, the document was old or not written properly, so it was a trial and error effort.

traceparent

In OpenCensus, it seems that the context was originally passed using header information such as X-B3-TraceId / X-B3-ParentSpanId, but now W3C standardized Trace Context .io / trace-context /) is used. This embeds a parameter called traceparent in the HTTP header and assembles" Trace ID "," Span ID "and" Trace Options (trace-flags) "based on it.

traceparent has the following values.

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

The code below is to parse traceparent and create SpanContext with OpenCensus.

int VERSION_SIZE = 2;
int TRACEPARENT_DELIMITER_SIZE = 1;
int TRACE_ID_HEX_SIZE = 2 * TraceId.SIZE;
int SPAN_ID_HEX_SIZE = 2 * SpanId.SIZE;
int TRACE_ID_OFFSET = VERSION_SIZE + TRACEPARENT_DELIMITER_SIZE;
int SPAN_ID_OFFSET = TRACE_ID_OFFSET + TRACE_ID_HEX_SIZE + TRACEPARENT_DELIMITER_SIZE;
int TRACE_OPTION_OFFSET = SPAN_ID_OFFSET + SPAN_ID_HEX_SIZE + TRACEPARENT_DELIMITER_SIZE;

// Get tranceparent
String traceparent = headers.getRequestHeaders().getFirst("traceparent");

// Parse traceparent
TraceId traceId = TraceId.fromLowerBase16(traceparent, TRACE_ID_OFFSET);
SpanId spanId = SpanId.fromLowerBase16(traceparent, SPAN_ID_OFFSET);
TraceOptions traceOptions = TraceOptions.fromLowerBase16(traceparent, TRACE_OPTION_OFFSET);

//Creating SpanContext from traceparent
SpanContext spanContext = SpanContext.create(traceId, spanId, traceOptions);

JAX-RS client (give traceparent when REST Call)

In order to convey the context to the remote API, it is necessary to add traceparent to the HTTP header etc. at the time of REST Call. You can do it manually, but the OpenCensus library for JAX-RS "opencensus-contrib-http-jaxrs ”Is used.

var url = 'http://localhost:5000';
var target = ClientBuilder.newClient()
        .target(url)
        .path("/account/" + userId + "/balance");

target.register(JaxrsClientFilter.class);
return target
        .request(MediaType.APPLICATION_JSON)
        .get(new GenericType<Map<String, Long>>() {});

By registering JaxrsClientFilter with target.register (JaxrsClientFilter.class), traceparent is automatically generated from the current Span and added to the request header.

JAX-RS client (creating Span from traceparent)

This is the implementation of api-core on the server side of JAX-RS. It seems that it can be set automatically by using the Container Filter of opencensus-contrib-http-jaxrs. It didn't work for some reason, so I'll implement it myself.

private static final TextFormat textFormat = Tracing.getPropagationComponent().getTraceContextFormat();
private static final TextFormat.Getter<HttpServletRequest> getter = new TextFormat.Getter<HttpServletRequest>() {
    @Override
    public String get(HttpServletRequest httpRequest, String s) {
        return httpRequest.getHeader(s);
    }
};

@GET
@Path("/{userId}/balance")
public Map<String, Long> getBalance(@Context HttpServletRequest request, @PathParam("userId") String userId) throws SpanContextParseException {
    var spanContext = textFormat.extract(request, getter);

    var depth = 1;
    var className = Thread.currentThread().getStackTrace()[depth].getClassName();
    var methodName = Thread.currentThread().getStackTrace()[depth].getMethodName();

    try (var ss = Tracing.getTracer()
            .spanBuilderWithRemoteParent(className + "$" + methodName, spanContext)
            .setRecordEvents(true)
            .setSampler(Samplers.alwaysSample())
            .startScopedSpan()) {
        var balance = service.getBalance(userId);
        return balance;
    }
}

By utilizing the TextFormat class, you can create aSpanContext without having to do the troublesome parsing yourself. Also, unlike normal Span creation, use spanBuilderWithRemoteParent to create a span from a remote context. There is no problem with spanBuilder when creating subsequent child Spans.

This completes the settings for distributed tracing using OpenCensus in Java, so please actually run it and check if it is linked to Jaeger or Cloud Trace.

Summary

I implemented distributed tracing using OpenCensus in Java. To be honest, there are documents at a glance and it is a famous specification, so there will be many blog articles, so I thought that it would be an easy win in about an hour, but I got hooked on the swamp and it took about an hour.

I feel like I want to get rid of the lack of documentation and deficiencies, but I'm sure that effort should be spent on Open Telemetry. Also, I think that the reason why the details are not written is that basically some people who are familiar with distributed tracing design the library and FW and then use it. MP OpenTracing or OpenTelemetry OpenTelemetry Auto-Instrumentation for Java is touched even if you are not familiar with it.

Well, as a result, I think I got more detailed this time, so I wonder if that was all right. Open Telemetry will have to be re-challenged soon.

Then Happy Hacking!

reference

• OpenCensus: QuickStart - Java
• Github: OpenCensus - Java -Translation of W3C Trace Context
• Distributed tracing with Opencensus and Jaeger in Java