Red Hat

Hawkular Blog

Grafana: new query interface

24 July 2017, by Joel Takvorian

There have been improvements lately in the Hawkular Grafana datasource that are worth mentioning.

  • The way to query metrics by tags has changed since plugin v1.0.8. It now takes advantage of the Hawkular Metrics' tags query language, that was introduced server-side in Metrics 0.24.0 and enhanced in 0.25.0. To sum it up, Metrics integrates a parser that allows queries such as: tag1 = 'awesome!' AND tag2 NOT IN ['foo', 'bar'].

  • The datasource in now able to fetch bucketized metrics stats, instead of raw metrics. It consists in aggregating datapoints in slices of time (buckets) and providing, for each slice, some statistics like min, max, average and more. The exact content of a bucket is described in Metrics REST API. Hawkular has always been able to provide metric stats server-side, but being able to use them in the Grafana plugin is new, introduced in v1.0.9.

New query interface

The new query interface

Tags query language

The first change is that you don’t have to choose between query by tag and query by metric id anymore, you can do both at the same time. Querying by tag will refine the available list of metric names (much like a filter) and can result in multiple metrics from a single query. By selecting a metric name, you restrict the query to only display that one. This filtering is really nice when there’s tons of metrics available, like in the case of hundreds of OpenShift pods being monitored with the same tenant.

The simple key/value pairs interface is now replaced with a more elaborated query builder, following the pattern: 'tag-key' 'operator' 'tag-value(s)' ['AND/OR' etc.]

The following images show a walk-through:

Selecting tag key

Selecting the tag key

Selecting tag operator

Selecting the tag operator

Selecting tag value

Selecting the tag value

The text fields include dynamic suggestions, you can use Grafana template variables within tag values, or enter free text. Once you’ve set up a tag query expression, the relevant metrics immediately show up on the chart and the list of available metrics in the dropdown list in updated.

Filtered metrics

Filtered metrics

This query builder lets you build almost any tag query that the Hawkular server understands. There are however some corner cases. For now this builder doesn’t allow you to prioritize expressions with parentheses. For instance, you cannot build c1 = 'foo' OR (b1 != 'B' AND a1 = 'abcd'). As a workaround you can turn off the query builder and directly type in your query expression.

Toggle editor mode

Toggle editor mode

It will be sent as is to the Hawkular Metrics server. This will also be useful to fill the gap if the language evolves server-side and this plugin isn’t updated immediately.

Stats query

The other important feature is the ability to run stats queries against Hawkular Metrics, instead of raw queries. There are several reasons to do this:

  • it reduces the network load, and client-side processing load, especially when raw data would contain tons of datapoints

  • it enables some aggregation methods

  • it also allows higher-level analysis with stats such as percentiles

To enable it, just clear the raw checkbox.

Toggle stats mode

Toggle stats mode

When you clear the raw checkbox, you can configure Multiple series aggregation to None, Sum or Average and can configure Stat type as avg, min, max, median, sum and different percentiles. You can display several different Stat types within the same query.

Stats without aggregation

Stats without aggregation: each two metrics show avg, min and max

Stats avg

Same query with series aggregation: the two metrics are averaged into one, which shows avg, min and max

If the query returns multiple series, use Multiple series aggregation to define if and how to aggregate them. None will show them individually on the chart. But consider for instance the case of an OpenShift pod with many replicas, and you’re tracking their memory usage. It may be more relevant here to aggregate all of them, both sum and average are meaningful here.

The Stat type option refers to an aggregation at a different level: not between multiple metrics, but within a single metric, all raw datapoints are aggregated within time buckets.

Conclusion

These two improvements aim a common goal, that is facilitating querying over large amounts of data. This is becoming crucial especially in the context of microservices and applications running on container platforms, as the number of metrics explodes. Proper metrics tagging is the corner stone to make sense of this data.





Protecting Jaeger UI with a sidecar security proxy

18 July 2017, by Juraci Paixão Kröhling

In a production deployment of Jaeger, it may be advantageous to restrict access to Jaeger’s Query service, which includes the UI. For instance, you might have internal security requirements to allow only certain groups to access trace data, or you might have deployed Jaeger into a public cloud. In a true microservices way, one possible approach is to add a sidecar to the Jaeger Query service, acting as a security proxy. Incoming requests hit our sidecar instead of reaching Jaeger’s Query service directly and the sidecar would be responsible for enforcing the authentication and authorization constraints.

Jaeger login screen

Incoming HTTP requests arrive at the route ①, which uses the internal service ② to resolve and communicate with the security proxy ③. Once the request is validated and all security constraints are satisfied, the request reaches Jaeger ④.

For demonstration purposes we’ll make use of Keycloak as our security solution, but the idea can be adapted to work with any security proxy. This demo should also work without changes with Red Hat SSO. For this exercise, we’ll need:

  • A Keycloak (or Red Hat SSO) server instance running. We’ll call its location ${REDHAT_SSO_URL}

  • An OpenShift cluster, where we’ll run Jaeger backend components. It might be as easy as oc cluster up

  • A local clone of the Jaeger OpenShift Production template

Note that we are not trying to secure the communication between the components, like from the Agent to the Collector. For this scenario, there are other techniques that can be used, such as mutual authentication via certificates, employing istio or other similar tools.

Preparing Keycloak

For this demo, we’ll run Keycloak via Docker directly on the host machine. This is to stress that Keycloak does not need to be running on the same OpenShift cluster as our Jaeger backend.

The following command should start an appropriate Keycloak server locally. If you already have your own Keycloak or Red Hat SSO server, skip this step.

docker run --rm --name keycloak-server -e KEYCLOAK_USER=admin -e KEYCLOAK_PASSWORD=password -p 8080:8080 jboss/keycloak

Once the Keycloak server is up and running, let’s create a realm for Jaeger:

  1. Login into Keycloak (http://<YOUR_IP>:8080/auth/admin/master/console) with admin as username and password as password

  2. In the top left corner, mouse over the Select realm box and click Add realm. Name it jaeger and click Create

  3. On Clients, click Create and set proxy-jaeger as the name and save it

  4. Set the Access Type to confidential and * as Valid Redirect URIs and save it. You might want to fine tune this in a production environment, otherwise you might be open to an attack known as "Unvalidated Redirects and Forwards".

  5. Open the Installation tab and select Keycloak OIDC JSON and copy the JSON that is shown. It should look like this, but the auth-server-url and secret will have different values.

{
  "realm": "jaeger",
  "auth-server-url": "http://192.168.2.111:8080/auth",
  "ssl-required": "external",
  "resource": "proxy-jaeger",
  "credentials": {
    "secret": "7f201319-1dfd-43cc-9838-057dac439046"
  }
}

And finally, let’s create a role and a user, so that we can log into Jaeger’s Query service:

  1. Under the Configure left-side menu, open the Roles page and click Add role

  2. As role name, set user and click Save

  3. Under the Manage left-side menu, open the Users page and click Add user

  4. Fill out the form as you wish and set Email verified to ON and click on Save

  5. Open the Credentials tab for this user and set a password (temporary or not).

  6. Open the Role mappings tab for this user, select the role user from the Available Roles list and click Add selected

Preparing OpenShift

For this demo, we assume you have an OpenShift cluster running already. If you don’t, then you might want to check out tools like minishift. If you are running a recent version of Fedora, CentOS or Red Hat Enterprise Linux you might want to install the package origin-clients and run oc cluster up --version=latest. This should get you a basic OpenShift cluster running locally.

To make it easier for our demonstration, we’ll add cluster-admin rights to our developer user and we’ll create the Jaeger namespace:

oc login -u system:admin
oc new-project jaeger
oc adm policy add-cluster-role-to-user cluster-admin developer -n jaeger
oc login -u developer

Preparing the Jaeger OpenShift template

We’ll use the Jaeger OpenShift Production template as the starting point: either clone the entire repository, or just get a local version of the template.

The first step is to add the sidecar container to the query-deployment object. Under the containers list, after we specify the jaeger-query, let’s add the sidecar:

        - image: jboss/keycloak-proxy
          name: ${JAEGER_SERVICE_NAME}-query-security-proxy
          volumeMounts:
          - mountPath: /opt/jboss/conf
            name: security-proxy-configuration-volume
          ports:
          - containerPort: 8080
            protocol: TCP
          readinessProbe:
            httpGet:
              path: "/"
              port: 8080

Note that container specifies a volumeMount named security-proxy-configuration-volume: we’ll use it to store the proxy’s configuration file. You should add the volume under the spec/template/spec node for query-deployment, sibling to the dnsPolicy property (it’s probably right under the previous code snippet):

        volumes:
          - configMap:
              name: ${JAEGER_SERVICE_NAME}-configuration
              items:
                - key: proxy
                  path: proxy.json
            name: security-proxy-configuration-volume

Now, we need to specify the ConfigMap, with the proxy’s configuration entry. To do that, we add a new top-level item to the template. As a suggestion, we recommend keeping it close to where it’s consumed. For instance, right before the query-deployment:

- apiVersion: v1
  kind: ConfigMap
  metadata:
    name: ${JAEGER_SERVICE_NAME}-configuration
    labels:
      app: jaeger
      jaeger-infra: security-proxy-configuration
  data:
    proxy: |
      {
          "target-url": "http://localhost:16686",
          "bind-address": "0.0.0.0",
          "http-port": "8080",
          "applications": [
              {
                  "base-path": "/",
                  "adapter-config": {
                    "realm": "jaeger",
                    "auth-server-url": "${REDHAT_SSO_URL}",
                    "ssl-required": "external",
                    "resource": "proxy-jaeger",
                    "credentials": {
                      "secret": "THE-SECRET-FROM-INSTALLATION-FILE"
                    }
                  }
            ,
            "constraints": [
                      {
                          "pattern": "/*",
                          "roles-allowed": [
                              "user"
                          ]
                      }
                  ]
              }
          ]
      }

Note that we are only allowing users with the role user to log into our Jaeger UI. In a real world scenario, you might want to adjust this to fit your setup. For instance, your user data might come from LDAP, and you only want to allow users from specific LDAP groups to access the Jaeger UI.

The secret within the credentials should match the secret we got from Keycloak at the beginning of this exercise. Our most curious readers will note that we mentioned the template parameter REDHAT_SSO_URL under the property auth-server-url. Either change that to your Keycloak server, or let’s specify a template parameter, allowing us to set this at deployment time. Under the parameters section of the template, add the following property:

- description: The URL to the Red Hat SSO / Keycloak server
  displayName: Red Hat SSO URL
  name: REDHAT_SSO_URL
  required: true
  value: http://THE-URL-FROM-THE-INSTALLATION-FILE:8080/auth
This value should be a location that is reacheable by both your browser and by the sidecar, like your host’s LAN IP (192.x, 10.x). Localhost/127.x is not going to work.

As a final step, we need to change the service to direct requests to the port 8080 (proxy) instead of 16686. This is done by changing the property targetPort on the service named query-service, setting it to 8080:

- apiVersion: v1
  kind: Service
  metadata:
    name: ${JAEGER_SERVICE_NAME}-query
    labels:
      app: jaeger
      jaeger-infra: query-service
  spec:
    ports:
    - name: jaeger-query
      port: 80
      protocol: TCP
      targetPort: 8080
    selector:
      jaeger-infra: query-pod
    type: LoadBalancer

As a reference, here’s the complete template file that can be used for this blog post.

Deploying

Now that we have everything ready, let’s deploy Jaeger into our OpenShift cluster. Run the following command from the same directory you stored the YAML file from the previous steps, referenced here by the name jaeger-production-template.yml:

oc process -f jaeger-production-template.yml | oc create -n jaeger -f -

During the first couple of minutes, it’s OK if the pods jaeger-query and jaeger-collector fail, as Cassandra will still be booting. Eventually, the service should be up and running, as shown in the following image.

Pod with sidecar on OpenShift

Once it is ready to serve requests, click on URL for the route (https://jaeger-query-jaeger.127.0.0.1.nip.io). You should be presented with a login screen, served by the Keycloak server. Login with the credentials you set on the previous steps, and you should reach the regular Jaeger UI.

Conclusion

In this exercise, we’ve seen how to add a security proxy to our Jaeger Query pod as a sidecar. All incoming requests go through this sidecar and all features available in Keycloak can be used transparently, such as 2-Factor authentication, service accounts, single sign-on, brute force attack protection, LDAP support and much more.





OpenTracing JAX-RS Instrumentation

10 July 2017, by Pavol Loffay

In the previous demo we have demonstrated how to instrument a Spring Boot app using OpenTracing, a vendor-neutral standard for distributed tracing. In this article we are going to instrument a Java API for RESTful Web Services (JAX-RS), and show you how to trace the business layer and add custom data to the trace.

Demo application

Creating a JAX-RS app from scratch can be a time consuming task, therefore in this case we are going to use Wildfly Swarm’s app generator. Select JAX-RS and CDI dependencies and hit generate button.

wf swarm generator
Figure 1: Wildfly Swarm generator.

The generated application contains one REST endpoint which returns hello world string. This endpoint is accessible on http://localhost:8080/hello. In the next step we are going to add instrumentation and simple business logic.

Instrumentation

Adding OpenTracing instrumentation to JAX-RS is very simple, just include the following dependency in the classpath and the tracing feature will be automatically registered.

<dependency>
    <groupId>io.opentracing.contrib</groupId>
    <artifactId>opentracing-jaxrs2</artifactId>
</dependency>

OpenTracing is just an API, therefore it is required to register a specific tracer instance. In this demo we are going to use Jaeger tracing system. The tracer should be created and initialized only once per process, hence ServletContextListener is the ideal place for this task:

@WebListener
public class TracingContextListener implements ServletContextListener {

  @Inject
  private io.opentracing.Tracer tracer;

  @Override
  public void contextInitialized(ServletContextEvent sce) {
    GlobalTracer.register(tracer);
  }

  @Override
  public void contextDestroyed(ServletContextEvent sce) {}

  @Produces
  @Singleton
  public static io.opentracing.Tracer jaegerTracer() {
    return new Configuration("wildfly-swarm", new Configuration.SamplerConfiguration(
        ProbabilisticSampler.TYPE, 1),
        new Configuration.ReporterConfiguration())
        .getTracer();
  }
}

Tracer initialization code requires to specify app name, which is in this case wildfly-swarm and sampler configuration.

Note that we are suing Java’s Context and Dependency Injection (CDI) to share a tracer instance in our app. If we forget to register a specific tracer instance, then the tracing feature would use NoopTracer. Now we can verify tracing by starting Jaeger server using the following command: docker run --rm -it --network=host jaegertracing/all-in-one and accessing the endpoint at http://localhost:8080/hello. Our trace with one span should be present in the UI at http://localhost:16686.

Instrumenting business logic

JAX-RS instrumentation provides nice visibility into your app, however, it is often necessary to add custom data to the trace to see what is happening in the service or database layer.

The following code snippet shows how the service layer can create and add data to the trace:

public class BackendService {

  @Inject
  private io.opentracing.Tracer tracer;

  public String action() throws InterruptedException {
    int random = new Random().nextInt(200);

    try (ActiveSpan span = tracer.buildSpan("action").startActive()) {
      anotherAction();
      Thread.sleep(random);
    }

    return String.valueOf(random);
  }

  private void anotherAction() {
    tracer.activeSpan().setTag("anotherAction", "data");
  }

Note that it’s not necessary to manually pass a span instance around. The method anotherAction accesses the current active span from the tracer.

With the additional instrumentation shown above, an invocation of the REST endpoint would result in a trace consisting of two spans, one representing the inbound server request, and the other the business logic. The span representing server processing is automatically considered as the parent for span created in business layer. If we created span in anotherAction then its parent would be span created in action method.

swarm jaeger
Figure 1: Jaeger showing reported spans.

Video

Conclusion

We have demonstrated that instrumenting a JAX-RS app is just a matter of adding a dependency and registering a tracer instance. If we would like to use a different OpenTracing implementation, Zipkin for instance, it would just require changing tracer producer code. No changes to the application or business logic! In the next demo we will wire this app with Spring Boot created in previous demo and deploy them on Kubernetes.





Using OpenTracing to collect Application Metrics in Kubernetes

26 June 2017, by Gary Brown

This article will show how OpenTracing instrumentation can be used to collect Application Metrics, in addition to (but independent from) reported tracing data, from services deployed within Kubernetes. These Application Metrics can then be displayed in your monitoring dashboard and used to trigger alerts.

2017 06 26 grafana error ratio

The example application

In a recent article we showed how a Spring Boot application could easily be instrumented using OpenTracing.

The example we are going to use in this article uses the same approach to create two services, ordermgr and accountmgr.

accountmgr presents a single REST endpoint (/getAccount) for internal use by ordermgr. The code for this endpoint is:

Account Managers’s Controller:
    @RequestMapping("/account")
    public String getAccount() throws InterruptedException {
        Thread.sleep(1 + (long)(Math.random()*500)); (1)
        if (Math.random() > 0.8) { (2)
            throw new RuntimeException("Failed to find account");
        }
        return "Account details";
    }
1 This line simply introduces a random delay, to make the collected metrics more interesting.
2 These three lines randomly cause an exception which will result in the span (associated with the REST endpoint invocation) being tagged as an error with associated log events identifying the error details.

ordermgr presents three REST endpoints for use by an end user. These are:

Order Manager’s Controller:
    @Autowired
    private io.opentracing.Tracer tracer; (1)

    @RequestMapping("/buy")
    public String buy() throws InterruptedException {
        Thread.sleep(1 + (long)(Math.random()*500)); (2)
        tracer.activeSpan().setBaggageItem("transaction", "buy"); (3)
        ResponseEntity<String> response = restTemplate.getForEntity(accountMgrUrl + "/account", String.class);
        return "BUY + " + response.getBody();
    }

    @RequestMapping("/sell")
    public String sell() throws InterruptedException {
        Thread.sleep(1 + (long)(Math.random()*500)); (2)
        tracer.activeSpan().setBaggageItem("transaction", "sell"); (3)
        ResponseEntity<String> response = restTemplate.getForEntity(accountMgrUrl + "/account", String.class);
        return "SELL + " + response.getBody();
    }

    @RequestMapping("/fail")
    public String fail() throws InterruptedException {
        Thread.sleep(1 + (long)(Math.random()*500)); (2)
        ResponseEntity<String> response = restTemplate.getForEntity(accountMgrUrl + "/missing", String.class); (4)
        return "FAIL + " + response.getBody();
    }
1 The service injects the OpenTracing Tracer to enable access to the active span.
2 All three methods introduce a random delay.
3 The buy and sell methods additionally set a baggage item transaction with the name of the business transaction being performed (i.e. buy or sell). For those not familiar with OpenTracing, the baggage concept allows information to be carried in band with the trace context between invoked services. We will show you how a baggage item can be used to isolate the metrics relevant only for a particular business transaction.
4 Invoking a non-existent endpoint on accountmgr will lead to an error being reported in the trace and metric data.

Adding Metrics Reporting to the OpenTracing instrumentation

The OpenTracing API defines the concept of a Span which represents a unit of work performed by a service, e.g. to receive a service invocation, perform some internal task (e.g. accessing a database) or invoking an external service. They provide an ideal basis upon which to report metrics (count and duration) regarding these points within a service.

Therefore a new OpenTracing contrib project has been established (initially just for Java) to intercept the finished spans, and create the relevant metrics. These metrics are then submitted to a MetricsReporter for recording - the initial implementation of this interface being for Prometheus.

The first step is to expose an endpoint for collecting the Prometheus metrics. Each service has the following configuration:

@Configuration
@ConditionalOnClass(CollectorRegistry.class)
public class PrometheusConfiguration {

     @Bean
     @ConditionalOnMissingBean
     CollectorRegistry metricRegistry() {
         return CollectorRegistry.defaultRegistry;
     }

     @Bean
     ServletRegistrationBean registerPrometheusExporterServlet(CollectorRegistry metricRegistry) {
           return new ServletRegistrationBean(new MetricsServlet(metricRegistry), "/metrics");
     }
}

This will allow the Prometheus metrics to be obtained from the service’s /metrics REST endpoint.

Each service then requires a configuration to obtain the io.opentracing.Tracer:

@Configuration
public class TracerConfiguration implements javax.servlet.ServletContextListener {

	@Bean
	public io.opentracing.Tracer tracer() {
		return io.opentracing.contrib.metrics.Metrics.decorate(
			io.opentracing.contrib.tracerresolver.TracerResolver.resolveTracer(),
			PrometheusMetricsReporter.newMetricsReporter()
				.withBaggageLabel("transaction","n/a")
				.build());
	}

	@Override
	public void contextInitialized(javax.servlet.ServletContextEvent sce) {
		sce.getServletContext().setAttribute(io.opentracing.contrib.web.servlet.filter.TracingFilter.SKIP_PATTERN, Pattern.compile("/metrics"));
	}

	...

The first method uses the TracerResolver to provide a vendor neutral approach for accessing a Tracer. This tracer is then enhanced with the metrics capability using a PrometheusMetricsReporter. This metrics reporter is further configured to add a special label related to the baggage key transaction (discussed later).

By default, the Servlet OpenTracing integration will trace all REST endpoints. Therefore in the second method above we add an attribute that will inform the instrumentation to ignore the /metrics endpoint. Otherwise we will have tracing data reported each time Prometheus reads the metrics for the service.

Deploying on Kubernetes

The steps to set up an environment on Kubernetes is discussed in the example codebase. A summary of the steps is:

  • Start minikube

    minikube start
    minikube dashboard
  • Deploy Prometheus - using the Prometheus Operator project to capture metrics from the services

    kubectl create -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/bundle.yaml
    
    # Wait until pods are green, then add configuration to locate service monitors based on label "team: frontend":
    kubectl create -f https://raw.githubusercontent.com/objectiser/opentracing-prometheus-example/master/prometheus-kubernetes.yml
    
    # Wait until these pods are green, then get the URL from the following command and open in browser:
    minikube service prometheus --url
  • Deploy Jaeger - an OpenTracing compatible tracing system

    kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/all-in-one/jaeger-all-in-one-template.yml
    
    # Once pods are green, then get the Jaeger dashboard URL from the following command and open in a browser
    minikube service jaeger-query --url
  • For this article, we also deployed Grafana to display the metrics, although the Prometheus dashboard could be used. Once Grafana is installed:

    • Obtain the Prometheus server URL using minikube service prometheus --url

    • Configure a new Datasource named Prometheus of type Prometheus and specify the URL obtained from the previous command

    • Download the example dashboard using the following command and import it into Grafana

      wget https://raw.githubusercontent.com/objectiser/opentracing-prometheus-example/master/simple/GrafanaDashboard.json

Once they are all running, then the simple example with the two services can be deployed. For this you will need to clone the example code repo, and follow these instructions.

At this stage the Kubernetes dashboard would look like this:

2017 06 26 kubernetes dashboard
Figure 1: Kubernetes dashboard

The example code includes a script that loops, randomly invoking the three REST endpoints provided by ordermgr. Once some example requests have been created, you can view the tracing dashboard:

2017 06 26 traces
Figure 2: Jaeger tracing dashboard

Then you can select a specific trace instance and see further details:

2017 06 26 trace
Figure 3: Jaeger trace instance view

This shows that the trace instance has three spans, the first representing the receipt of the /buy request on ordermgr, the second where ordermgr is invoking accountmgr, and finally the accountmgr receiving the /hello request. In this particular trace instance, the accountmgr invocation has reported an error, indicated by the error=true tag.

Now we will look at the Grafana dashboard to see what metrics have been reported from the OpenTracing instrumentation within the two services:

2017 06 26 grafana dashboard
Figure 4: Grafana dashboard

This dashboard includes three graphs, the first showing the number of spans created (i.e. span count) by our sell() method, and we can use it to track how many times this business operation has been executed. The second showing the average duration of the spans, and third showing the ratio between successful and erronous spans.

The metrics reported by Prometheus are based on a range of labels - a metric exists for each unique combination of those labels.

The standard labels included with the OpenTracing java-metrics project are: operation, span.kind and error.

With this particular example, we also included the transaction label.

However when the services are deployed to Kubernetes, the following additional labels are included for free: pod, instance, service, job and namespace.

In our example Prometheus queries, we have ignored most of the Kubernetes added labels (except service) so that the metrics are aggregated across the specific pods, namespaces, etc. However, having these labels available means it is possible to segment the metrics in whatever way is required to analyse the data.

When using the java-metrics project outside of Kubernetes, it is still possible to include the service label, however you would configure this when setting up the tracer.

We can also filter the data, to focus on specific areas of interest:

2017 06 26 grafana txn service
Figure 5: Customized Grafana graph focusing on metrics for transaction 'sell' and service 'accountmgr'

In this image we have filtered the metrics based on the transaction='sell' and service='accountmgr'. This is where using the metric label based on the baggage item transaction can be useful, to understand the usage of a particular shared service by a business transaction. With further work it would be possible to show the distribution of requests for a service across the various business transactions.

Video

Conclusion

This article has shown how a service can be instrumented once (using OpenTracing) and generate both tracing and application metrics.

When deployed to a Kubernetes environment, the metrics also benefit from an additional set of labels automatically added by the infrastructure, describing the service, pod, namespace, etc. This makes it easy to isolate specific metrics of interest, or view high level aggregated metrics to gain an overview of your applications performance.





OpenTracing Spring Boot Instrumentation

13 June 2017, by Pavol Loffay

In this demo series we are going to look at how simple it is to instrument various Java frameworks using OpenTracing. You will see that it requires minimal changes to the application code. In the last demo we will have microservice apps deployed on Kubernetes and all services will be traced with an OpenTracing compliant tracing system.

In this first demo we are going to develop and trace a simple Spring Boot app.

Create a web application

First let’s write a simple web app. Or better, let’s generate it! All we have to do is just to select a web dependency.

spring initializr
Figure 1: Spring boot generator.

Now the application is generated, but it does not contain any web controller. We are going to implement a simple controller with two methods. One will return a greeting and the other creates HTTP request which calls hello endpoint. This demonstrates simple request chaining between services. Do not worry all the code is on GitHub. At the bottom of this article you will find all necessary links.

Hello Controller:
@RestController
public class HelloController {

    @Autowired
    private RestTemplate restTemplate;

    @RequestMapping("/hello")
    public String hello() {
        return "Hello from Spring Boot!";
    }

    @RequestMapping("/chaining")
    public String chaining() {
        ResponseEntity<String> response = restTemplate.getForEntity("http://localhost:8080/hello", String.class);
        return "Chaining + " + response.getBody();
    }
}

Now the application can serve requests for URLs http://localhost:8080/hello and http://localhost:8080/chaining. The app is still not instrumented, we won’t see any data coming to a tracing system.

Instrumentation

Instrumentation with OpenTracing integrations is very simple. For Spring Boot there is an auto-configuration which instruments all REST controllers and RestTemplate beans. Just add the following dependency to the classpath:

<dependency>
    <groupId>io.opentracing.contrib</groupId>
    <artifactId>opentracing-spring-web-autoconfigure</artifactId>
</dependency>

This dependency requires only one thing and that is a tracer bean which will be used to report data to the chosen tracing system. If we don’t specify this bean auto-configuration will choose NoopTracer.

Because we are using OpenTracing instrumentation we are not bound to any specific tracing system. We will now show how to first use Jaeger and then switch to Zipkin. We will see that changing the tracing system is just a matter of configuration.

As we mentioned the only tracing configuration needed here is to provide a tracer bean.

Jaeger

To create a Jaeger tracer is very simple. It just requires a sampler configuration and because it is a demo we are going to sample all requests. Note that we are not specifying the URL to Jaeger server. By default it will assume that it runs on localhost.

@Bean
public io.opentracing.Tracer jaegerTracer() {
    return new Configuration("spring-boot", new Configuration.SamplerConfiguration(ProbabilisticSampler.TYPE, 1),
        new Configuration.ReporterConfiguration())
        .getTracer();

Now we can start the Jaeger server using docker run --rm -it --network=host jaegertracing/all-in-one, compile and run our app. When everything is up and running generate some requests to URL’s defined in the previous section.

Open the Jaeger UI on http://localhost:16686:

boot jaeger traces
Figure 1: Jaeger showing reported traces.

On the picture we can see traces for the request to the /chaining endpoint. There are three spans: one representing server processing of /chaining, the second a client request to /hello and the third server processing of /hello endpoint.

Zipkin

Now let’s benefit from OpenTracing and switch tracing system with O(1) effort. To do that we just need to provide an instance of Zipkin tracer bean. Do not forget to comment out the Jaeger tracer bean, otherwise instrumentation would not know which tracer to use.

Zipkin configuration is very similar it just requires to know Zipkin URL:

@Bean
public io.opentracing.Tracer zipkinTracer() {
    OkHttpSender okHttpSender = OkHttpSender.create("http://localhost:9411/api/v1/spans");
    AsyncReporter<Span> reporter = AsyncReporter.builder(okHttpSender).build();
    Tracing braveTracer = Tracing.newBuilder().localServiceName("spring-boot").reporter(reporter).build();
    return BraveTracer.create(braveTracer);
}

Zipkin server can be started with docker run --rm -it -p 9411:9411 openzipkin/zipkin. Now we have to rebuild and start our demo app and generate requests.

boot zipkin traces
Figure 1: Zipkin showing reported traces.

This screenshot also show traces for invocation of /chaining endpoint. In this case it shows only two spans because Zipkin uses a shared span model which means that client and server invocation of /hello uses the same span. This is a great example that shows how different OpenTracing providers might model and show things differently.

Video

Conclusion

We have seen how simple it is to instrument Spring Boot with OpenTracing. This instrumentation leverages all key OpenTracing benefits like: vendor-neutrality, O(1) change of tracing system or wiring different instrumentations together. In the next blog post we will look at JAX-RS instrumentation and in the last demo all applications will be deployed on Kubernetes and traced using Jaeger’s production deployment with Cassandra cluster.





Older posts:

RSS Feed

redhatlogo-white

© 2016 | Hawkular is released under Apache License v2.0