Distributed tracing with OpenCensus and Java

Introduction

I tried to implement distributed tracing with OpenCensus API and Java. OpenCensus is a specification based on Google's Cloud Trace (formerly StackDriver Trace) and is a specification for distributed tracing and metric collection. By using this, the PG side can link the backend to Jeager, OpenZipkin, Cloud Trace, etc. while using the unified API.

In fact, OpenCensus is already old and is now standardized as Open Telemetry by integrating with Open Tracing with similar specifications. This includes commercial APMs such as Datadog, NewRelic, Dynatrace, and Instana, so you should use this in the future. So, at first I was playing with Open Telemetry, but although I was able to cooperate with Jeager, it was Cloud Trace does not yet support Java version. For the time being, I tried OpenCensus, which seems to have a lot of documentation.

However, the documents were not organized as much as I expected, so I will summarize them as a reminder.

The code I created this time is below. https://github.com/koduki/miniban/tree/example/opensensus

System configuration

As a system configuration, it simply receives a request with ʻapi-endpoint and throws the process to the backend ʻapi-core with business logic with REST / JSON.

opencensus.png

It's not as subdivided as microservices, but it's a common configuration and I think it's good enough to verify distributed tracing. Each API is implemented in JAX-RS using Quarkus. Basically, we don't use MicroProfile-specific API, so any Java EE environment should work as it is.

Implementation

Dependent libraries

Add the following to pom.xml as a dependent library.

<dependency>
    <groupId>io.opencensus</groupId>
    <artifactId>opencensus-api</artifactId>
    <version>0.26.0</version>
</dependency>
<dependency>
    <groupId>io.opencensus</groupId>
    <artifactId>opencensus-impl</artifactId>
    <version>0.26.0</version>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>io.opencensus</groupId>
    <artifactId>opencensus-exporter-trace-stackdriver</artifactId>
    <version>0.26.0</version>
</dependency>
<dependency>
    <groupId>io.opencensus</groupId>
    <artifactId>opencensus-exporter-trace-jaeger</artifactId>
    <version>0.26.0</version>
</dependency>
<dependency>
    <groupId>io.opencensus</groupId>
    <artifactId>opencensus-contrib-http-jaxrs</artifactId>
    <version>0.26.0</version>
</dependency>

The APIs you need are opencensus-api and opencensus-impl. opencensus-exporter-xxx is a variety of exporter libraries that send trace information, and ʻopencensus-contrib-http-jaxrs` is a convenient library for JAX-RS. This time, I have specified two exporters, stackdriver and jaeger, but usually one of them.

Exporter initialization

First, initialize and register the Exporter. This only needs to be done once, so use @Initialized (ApplicationScoped.class) to load it at startup.

Bootstrap.java


@ApplicationScoped
public class Bootstrap {
    public void handle(@Observes @Initialized(ApplicationScoped.class) Object event) {
        JaegerTraceExporter.createAndRegister(
                JaegerExporterConfiguration.builder()
                        .setThriftEndpoint("http://localhost:14268/api/traces")
                        .setServiceName("api-endpoint")
                        .build());
    }
}

Quick Start takes the URL directly as an argument to createAndRegister, but this is already non-existent. It looks like the recommended code, so use JaegerExporterConfiguration.

As you can see, use setThriftEndpoint to specify the URL of the link destination, and use setServiceName to use the service name on Jaeger.

As long as you change the registered part of this Exporter, you can switch the backend arbitrarily. For example, if you want to use Cloud Trace, change as follows.

Bootstrap.java


@ApplicationScoped
public class Bootstrap {
    public void handle(@Observes @Initialized(ApplicationScoped.class) Object event) {
        String gcpProjectId = "your GCP project ID";
        StackdriverTraceExporter.createAndRegister(
                StackdriverTraceConfiguration.builder()
                        .setProjectId(gcpProjectId)
                        .build());
    }
}

Creating a Trace Span

Next is the creation of Span.

try (Scope ss = Tracing.getTracer()
        .spanBuilder("Span Name")
        .setRecordEvents(true)
        .setSampler(Samplers.alwaysSample())
        .startScopedSpan()) {

    // do somthing.
}

Tracing.getTracer () gets the tracer from a singleton or global. From there, use spanBuilder to assemble the Span. The point is setSampler. The sampler specifies how often to get the trace. If you do not specify Samplers.alwaysSample () here, the trace information will not always be written. In production, there may be cases where an appropriate threshold is set in terms of cost and load, but at the time of testing, there are too few requests other than load testing, so be sure to specify always. Details of other samplers can be found at here.

By the way, it is troublesome to write the above every time, so I defined the following helper somed.

public static <R> R trace(Supplier<R> callback) {
    var depth = 2;
    var className = Thread.currentThread().getStackTrace()[depth].getClassName();
    var methodName = Thread.currentThread().getStackTrace()[depth].getMethodName();

    try (var ss = Tracing.getTracer()
            .spanBuilder(className + "$" + methodName)
            .setRecordEvents(true)
            .setSampler(Samplers.alwaysSample())
            .startScopedSpan()) {
        return callback.get();
    }
}

The class name and method name are acquired and automatically specified in the Span name. When using it, it looks like the following.

@GET
@Path("/{userId}/balance")
public Map<String, Long> getBalance(@PathParam("userId") String userId) {
    return trace(() -> {
        var balance = service.getBalance(userId);
        return balance;
    });
}

Propagation of Trace Context

If you are in the same application, just do startScopedSpan as above and the nested Span will be created without permission. However, the heart of distributed tracing is inter-system cooperation. Therefore, it is necessary to link remote contexts. However, from this point on, the document was old or not written properly, so it was a trial and error effort.

traceparent

In OpenCensus, it seems that the context was originally passed using header information such as X-B3-TraceId / X-B3-ParentSpanId, but now W3C standardized Trace Context .io / trace-context /) is used. This embeds a parameter called traceparent in the HTTP header and assembles" Trace ID "," Span ID "and" Trace Options (trace-flags) "based on it.

traceparent has the following values.

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

The code below is to parse traceparent and create SpanContext with OpenCensus.

int VERSION_SIZE = 2;
int TRACEPARENT_DELIMITER_SIZE = 1;
int TRACE_ID_HEX_SIZE = 2 * TraceId.SIZE;
int SPAN_ID_HEX_SIZE = 2 * SpanId.SIZE;
int TRACE_ID_OFFSET = VERSION_SIZE + TRACEPARENT_DELIMITER_SIZE;
int SPAN_ID_OFFSET = TRACE_ID_OFFSET + TRACE_ID_HEX_SIZE + TRACEPARENT_DELIMITER_SIZE;
int TRACE_OPTION_OFFSET = SPAN_ID_OFFSET + SPAN_ID_HEX_SIZE + TRACEPARENT_DELIMITER_SIZE;

// Get tranceparent
String traceparent = headers.getRequestHeaders().getFirst("traceparent");

// Parse traceparent
TraceId traceId = TraceId.fromLowerBase16(traceparent, TRACE_ID_OFFSET);
SpanId spanId = SpanId.fromLowerBase16(traceparent, SPAN_ID_OFFSET);
TraceOptions traceOptions = TraceOptions.fromLowerBase16(traceparent, TRACE_OPTION_OFFSET);

//Creating SpanContext from traceparent
SpanContext spanContext = SpanContext.create(traceId, spanId, traceOptions);

JAX-RS client (give traceparent when REST Call)

In order to convey the context to the remote API, it is necessary to add traceparent to the HTTP header etc. at the time of REST Call. You can do it manually, but the OpenCensus library for JAX-RS "opencensus-contrib-http-jaxrs ”Is used.

var url = 'http://localhost:5000';
var target = ClientBuilder.newClient()
        .target(url)
        .path("/account/" + userId + "/balance");

target.register(JaxrsClientFilter.class);
return target
        .request(MediaType.APPLICATION_JSON)
        .get(new GenericType<Map<String, Long>>() {});

By registering JaxrsClientFilter with target.register (JaxrsClientFilter.class), traceparent is automatically generated from the current Span and added to the request header.

JAX-RS client (creating Span from traceparent)

This is the implementation of api-core on the server side of JAX-RS. It seems that it can be set automatically by using the Container Filter of opencensus-contrib-http-jaxrs. It didn't work for some reason, so I'll implement it myself.

private static final TextFormat textFormat = Tracing.getPropagationComponent().getTraceContextFormat();
private static final TextFormat.Getter<HttpServletRequest> getter = new TextFormat.Getter<HttpServletRequest>() {
    @Override
    public String get(HttpServletRequest httpRequest, String s) {
        return httpRequest.getHeader(s);
    }
};

@GET
@Path("/{userId}/balance")
public Map<String, Long> getBalance(@Context HttpServletRequest request, @PathParam("userId") String userId) throws SpanContextParseException {
    var spanContext = textFormat.extract(request, getter);

    var depth = 1;
    var className = Thread.currentThread().getStackTrace()[depth].getClassName();
    var methodName = Thread.currentThread().getStackTrace()[depth].getMethodName();

    try (var ss = Tracing.getTracer()
            .spanBuilderWithRemoteParent(className + "$" + methodName, spanContext)
            .setRecordEvents(true)
            .setSampler(Samplers.alwaysSample())
            .startScopedSpan()) {
        var balance = service.getBalance(userId);
        return balance;
    }
}

By utilizing the TextFormat class, you can create aSpanContext without having to do the troublesome parsing yourself. Also, unlike normal Span creation, use spanBuilderWithRemoteParent to create a span from a remote context. There is no problem with spanBuilder when creating subsequent child Spans.

This completes the settings for distributed tracing using OpenCensus in Java, so please actually run it and check if it is linked to Jaeger or Cloud Trace.

Summary

I implemented distributed tracing using OpenCensus in Java. To be honest, there are documents at a glance and it is a famous specification, so there will be many blog articles, so I thought that it would be an easy win in about an hour, but I got hooked on the swamp and it took about an hour.

I feel like I want to get rid of the lack of documentation and deficiencies, but I'm sure that effort should be spent on Open Telemetry. Also, I think that the reason why the details are not written is that basically some people who are familiar with distributed tracing design the library and FW and then use it. MP OpenTracing or OpenTelemetry OpenTelemetry Auto-Instrumentation for Java is touched even if you are not familiar with it.

Well, as a result, I think I got more detailed this time, so I wonder if that was all right. Open Telemetry will have to be re-challenged soon.

Then Happy Hacking!

reference

Recommended Posts

Distributed tracing with OpenCensus and Java
Install Java and Tomcat with Ansible
Use JDBC with Java and Scala.
Output PDF and TIFF with Java 8
Encrypt with Java and decrypt with C #
Monitor Java applications with jolokia and hawtio
Link Java and C ++ code with SWIG
Let's try WebSocket with Java and javascript!
[Java] Reading and writing files with OpenCSV
Java and JavaScript
XXE and Java
Build and test Java + Gradle applications with Wercker
Try to link Ruby and Java with Dapr
JSON with Java and Jackson Part 2 XSS measures
Prepare a scraping environment with Docker and Java
KMS) Envelope encryption with openssl and java decryption
Encrypt / decrypt with AES256 in PHP and Java
[Java] Convert and import file values with OpenCSV
[Review] Reading and writing files with java (JDK6)
Getters and setters (Java)
Change seats with java
Solving with Ruby and Java AtCoder ABC129 D 2D array
[Java] Thread and Runnable
Summary of ToString behavior with Java and Groovy annotations
Install Java with Ansible
Compile with Java 6 and test with Java 11 while running Maven on Java 8
Solving with Ruby, Perl and Java AtCoder ABC 128 C
Java true and false
[Java] String comparison and && and ||
Comfortable download with JAVA
[Java] Refer to and set private variables with reflection
Switch java with direnv
Java --Serialization and Deserialization
[Java] Arguments and parameters
Download Java with Ansible
I want to transition screens with kotlin and java!
timedatectl and Java TimeZone
[Java] Branch and repeat
Prepare the environment for java11 and javaFx with Ubuntu 18.4
Let's scrape with Java! !!
Face recognition app made with Amazon Rekognition and Java
Build Java with Wercker
[Java] Variables and types
java (classes and instances)
[Java] Development with multiple files using package and import
Serverless Java EE starting with Quarkus and Cloud Run
[Java] Overload and override
Endian conversion with JAVA
Find the address class and address type from the IP address with Java
Getting started with Java and creating an AsciiDoc editor with JavaFX
Window aggregation of sensor data with Apache Flink and Java 8
Handle Java 8 date and time API with Thymeleaf with Spring Boot
Log aggregation and analysis (working with AWS Athena in Java)
Handle exceptions coolly with Java 8 lambda expressions and Stream API
I want to make a list with kotlin and java!
I want to make a function with kotlin and java!
Kotlin post- and pre-increment and operator overload (comparison with C, Java, C ++)
Problems with Dijkstra's algorithm using PriorityQueue and adjacency list (java)
[Java] Check the difference between orElse and orElseGet with IntStream
Socket communication with a web browser using Java and JavaScript ②
Socket communication with a web browser using Java and JavaScript ①