Red Hat

Hawkular Blog

Processing Hawkular-Metrics data with Python Pandas

22 February 2017, by Michael Burman

Storing the time series is not worth much if we can’t actually do something valuable with it. For some time, many data analytics have used Python tools to extract valuable information from the available data. Today, lets look at one of the popular options, Pandas. In this blog post I will show how to fetch data from the Hawkular-Metrics and turn it to a processable form for the Pandas and do something with the data.

As a data, I’ve taken some real world values from my current air-to-water heat pump and how it has functioned in some non-ideal circumstances. To make things more difficult, I’ve started logging the information after there’s been historical data already so the starting values are not from the same point of time. To make things a bit more complicated I’ve also included some temperature values.

Lets start by introducing a helper function and initialize pandas. You can install Pandas with pip install pandas or with dnf or apt.

import numpy as np
import pandas as pd
from hawkular import *

c = HawkularMetricsClient(tenant_id='pandas_test', port=8080)

def read_hawkular(c, id):
    s = c.query_metric(MetricType.Gauge, id, fromEarliest='true')
    df2 = pd.DataFrame(s, columns=['timestamp', 'value'])
    df2['timestamp'] = pd.to_datetime(df2['timestamp'], unit='ms')
    df2 = df2.set_index('timestamp')
    df2 = df2.rename(columns={'value': id})
    return df2

Now, with the read_hawkular we can fetch data from the Hawkular-Metrics using the given id. I’ve hardcoded Gauge as the metric type in this case as that’s the type of the values we’re processing today. We’ll also make the id as the column name for these values and use the timestamp as our index. Pandas will automatically convert the values to float64 and with our hints use the timestamps as the indexing value.

Lets insert some data:

c.push(MetricType.Gauge, 'pollucom.reading', 41955, timestamp=1487192074052)
c.push(MetricType.Gauge, 'meter.reading', 584.22, timestamp=1487192074145)
c.push(MetricType.Gauge, 'temperature.reading', 0.0, timestamp=1487192075645)

c.push(MetricType.Gauge, 'pollucom.reading', 42007, timestamp=1487272174104)
c.push(MetricType.Gauge, 'meter.reading', 606.41, timestamp=1487272174467)
c.push(MetricType.Gauge, 'temperature.reading', 1.6, timestamp=1487272176067)

c.push(MetricType.Gauge, 'pollucom.reading', 42075, timestamp=1487403934003)
c.push(MetricType.Gauge, 'meter.reading', 636.60, timestamp=1487403934732)
c.push(MetricType.Gauge, 'temperature.reading', 0.7, timestamp=1487403932678)

c.push(MetricType.Gauge, 'pollucom.reading', 42234, timestamp=1487663134340)
c.push(MetricType.Gauge, 'meter.reading', 702.29, timestamp=1487663134230)
c.push(MetricType.Gauge, 'temperature.reading', -3.3, timestamp=1487663134678)

c.push(MetricType.Gauge, 'pollucom.reading', 42296, timestamp=1487745987030)
c.push(MetricType.Gauge, 'meter.reading', 732.32, timestamp=1487745987575)
c.push(MetricType.Gauge, 'temperature.reading', -8.8, timestamp=1487745987976)

As you can already see from the data, we use millisecond precision when fetching the data and the timestamps have some variance in them. Now, using the previously defined function we will fetch the data to a Pandas DataFrame:

# Read the data

pollu = read_hawkular(c, 'pollucom.reading')
meter = read_hawkular(c, 'meter.reading')
temperature = read_hawkular(c, 'temperature.reading')

# Lets rename the columns

temperature = temperature.rename(columns={'temperature.reading': 'temperature'})
meter = meter.rename(columns={'meter.reading': 'electricity'})
pollu = pollu.rename(columns={'pollucom.reading': 'heat'})

Three distinct series are now stored in three distinct DataFrames. Combining them and looking at the data reveals something interesting:

stats = pd.concat([pollu, meter, temperature]).sort_index()
>>> stats
                         electricity   heat  temperature
timestamp
2017-02-15 20:54:34.052          NaN  41955          NaN
2017-02-15 20:54:34.145       584.22    NaN          NaN
2017-02-15 20:54:35.645          NaN    NaN          0.0
2017-02-16 19:09:34.104          NaN  42007          NaN
2017-02-16 19:09:34.467       606.41    NaN          NaN
2017-02-16 19:09:36.067          NaN    NaN          1.6
2017-02-18 07:45:32.678          NaN    NaN          0.7
2017-02-18 07:45:34.003          NaN  42075          NaN
2017-02-18 07:45:34.732       636.60    NaN          NaN
2017-02-21 07:45:34.230       702.29    NaN          NaN
2017-02-21 07:45:34.340          NaN  42234          NaN
2017-02-21 07:45:34.678          NaN    NaN         -3.3
2017-02-22 06:46:27.030          NaN  42296          NaN
2017-02-22 06:46:27.575       732.32    NaN          NaN
2017-02-22 06:46:27.976          NaN    NaN         -8.8
>>>

Now we see a very common problem for the time series processing. All the timestamps are close to each other, but they’re not quite aligned. So we can’t read from the same row what statistics we had for each value. If we wanted to fill the values from previous rows, we could do that easily with fillna function of the Pandas. However, in this case it makes more sense to resample the data to a one-day buckets and then drop the duplicate rows:

stats = stats.resample('D', fill_method='ffill').drop_duplicates()

>>> stats
            electricity   heat  temperature
timestamp
2017-02-15       584.22  41955          0.0
2017-02-16       606.41  42007          1.6
2017-02-18       636.60  42075          0.7
2017-02-21       702.29  42234         -3.3
2017-02-22       732.32  42296         -8.8
>>>

That’s much better. Now we can easily see the statistics for each day and how the counters had behaved. In this example it does not matter that we’re missing data, but there’s lots of tools in the Pandas to generate that missing data based on the rows we have. In this example, lets look at how to calculate from the values our coefficiency ratio (amount of heat generated divided by electricity used). We’ll do this by taking the difference of previous row to the current row from both heat as well as electricity:

stats['cop'] = stats.diff().apply(lambda row: row['heat'] / row['electricity'], axis=1)

>>> stats
            electricity   heat  temperature       cop
timestamp
2017-02-15       584.22  41955          0.0       NaN
2017-02-16       606.41  42007          1.6  2.343398
2017-02-18       636.60  42075          0.7  2.252401
2017-02-21       702.29  42234         -3.3  2.420460
2017-02-22       732.32  42296         -8.8  2.064602
>>>

The first row shows NaN as we don’t have any information prior to it for calculation. The coefficiency numbers show something worrysome, they don’t seem to follow a pattern normally associated with them (higher temperature gives higher coefficiency). For a simple check, lets check the Pearson correlation between avg temperature and coefficiency:

>>> stats.temperature.corr(stats.cop)
0.65064171188572928
>>>

I’ll leave the interpretation to the viewer, but lets comment quickly that the number of datapoints is too low in this example to have a meaningful value.

Conclusion

This was a short example on how to get started with Pandas and Hawkular-Metrics, but even with it we were able to calculate how efficiently our heat pump was working and actually notice that it’s working sub-optimally when temperature is high in this small sample scenario. The Pandas library gives multiple tools to process time series to a form that they can be used for further analytical processing, including but not limited to resampling, shifting, frequency conversions and periodic calculations. After modifying the data it can be easily further processed with other Python tools and even sent back to the Hawkular-Metrics in the processed form for storage purposes.





Hawkular Services 0.32.Final

15 February 2017, by Heiko W. Rupp

Hawkular services is a ready to run distribution of Hawkular-Metrics with Alerts, Inventory, the WildFly agent and other components. The version 0.32.0.Final has just been released and is ready to be tested and integrated with other projects.

What’s in this release

This release includes those changes:

  • Official docker images (see below)

  • Agent version 0.27

  • Agent has a new flag 'immutable', that when set will prevent modifications of its config or running state altering operations. That flag can be set via -Dhawkular.agent.immutable=true

  • Hawkular-Metrics version 0.24.1

  • Better support for running on OpenShift

The following video gives a walk-through of some of the features

Hawkular Services is released every week on Tuesdays.

Get started

To get started with Hawkular Services, download the latest release, unzip it, add a user and set the Agent to use the credentials for the user just added.

It can be accomplished with the following commands:

export HAWKULAR_HOME="/path/to/hawkular-services"
export HAWKULAR_USERNAME="jdoe"
export HAWKULAR_PASSWORD="password"

cd "${HAWKULAR_HOME}"
"${HAWKULAR_HOME}/bin/add-user.sh" \
  -a \
  -u "${HAWKULAR_USERNAME}" \
  -p "${HAWKULAR_PASSWORD}" \
  -g read-write,read-only
"${HAWKULAR_HOME}/bin/standalone.sh" -Dhawkular.rest.user="${HAWKULAR_USERNAME}" -Dhawkular.rest.password="${HAWKULAR_PASSWORD}"

Before starting the server, you need to have Cassandra 3.0.9 up and running with the RPC port enabled either via setting the env variable of CASSANDRA_START_RPC to true

or by editing cassandra.yml and setting it there:

# Whether to start the thrift rpc server.
start_rpc: true

Using ccm you can use this to update the config: ccm updateconf 'start_rpc: true'. Another option is via nodetool enablethrift.

Using Postgres

To use Postgres as backend you need to pass additional properties to the standalone.sh command above:

"${HAWKULAR_HOME}/bin/standalone.sh" -Dhawkular.rest.user="${HAWKULAR_USERNAME}" \
    -Dhawkular.rest.password="${HAWKULAR_PASSWORD} \
    -Dhawkular.inventory.sql.jdbc.url=jdbc:postgresql://1.2.3.4:5432/hawkular \
    -Dhawkular.inventory.sql.jdbc.username=jdoe \
    -Dhawkular.inventory.sql.jdbc.password=password

To use a postgres database called hawkular on host 1.2.3.4 owned by a user jdoe with a password of password.

Build from master and Cassandra

If you build from master, you can pass -Pembeddedc to get a distribution with embedded Cassandra for local development. Similarly if you build via -Pdev a default user of jdoe/password will be installed and also be used with the agent.

Use Grafana with Hawkular-services

We do now have a Grafana datasource to make it easier to connect to Hawkular-services from Grafana. You can download it from Grafana.net. The website of the datasource tells more.

Get started via Docker builds

There are official Docker images of the release available on DockerHub at Hawkular services

The latest tag will always point to the latest release.

NOTE

There are also images with a devel tag, which are pushed by internal CI on individual commits.

Unofficial WildFly + agent

There is also an instrumented WildFly server available at Instrumented WildFly.

Both images have been instrumented with the jdoe/password user for the Hawkular server and the agent.

See also http://pilhuhn.blogspot.de/2016/06/using-hawkular-services-via-docker.html on some more details about using those images.





Monitoring Canary Releases with Hawkular APM

13 February 2017, by Juraci Paixão Kröhling

In our last release, we built a Services screen on top of a feature added a couple of releases ago, the "service name" and "build stamps". These properties can be specified as an environment variable or system property, and are automatically detected when the monitored application runs on OpenShift.

Bars comparing performance of different service versions

In this blog post, we’ll show you how to use that screen to monitor a Canary release scenario. A Canary release is a technique used to minimize downtime (and risk) between deployments by preparing the new version on a completely different production environment and, once this new version is ready to serve requests, redirect some traffic from the "old" version to the "new" version. If this new version looks good, it gradually replaces the old version, eventually taking over all traffic. This technique builds on top of the Blue-Green deployment technique.

Getting the example running

We’ll use our Vert.x OpenTracing example for this demo. First, let’s deploy the example into OpenShift, along with Hawkular APM, by running the following Ansible playbook:

$ ansible-playbook vertx-opentracing.yml

After a few minutes, it should print out the address for the Order Manager microservice, like this:

TASK [Print out the order-manager address] *************************************
ok: [localhost] => {
    "msg": "The hostname for the Order Manager service is: http://order-manager-order-manager.192.168.2.107.xip.io"
}

At this point, change the genorders.sh script, to use this host instead of the original http://localhost:8180 on the curl command. It should then read: curl -X POST -H "Content-Type: application/json" -d "$data" http://order-manager-order-manager.192.168.2.107.xip.io/orders. Note that your hostname might be different than our example. You can then run the genorders.sh script, which will simulate some random requests to the Order Manager. The script will execute until it’s cancelled with Ctrl+C, and should have an output similar to this:

Creating an order for account=brian item=laptop quantity=3
{
  "accountId" : "brian",
  "itemId" : "laptop",
  "quantity" : 3,
  "id" : "d3f09a97-6f87-45c5-98c8-b592b6beb501"
}

Creating an order for account=sarah item=car quantity=4
Not account found

Creating an order for account=steve item=car quantity=2
{
  "accountId" : "steve",
  "itemId" : "car",
  "quantity" : 2,
  "id" : "7f095b1e-94a2-4398-8480-01aaea43e021"
}

Hawkular APM should be located under the service name hawkular-apm, so, the URL follows the pattern: https://hawkular-apm-order-manager.YOUR-IP.xip.io. Note that you’ll need to change the YOUR-IP part.

After a few minutes with our genorders.sh script running, the following can be seen:

Hawkular APM Dashboard with Data

Let’s open the Services tab, select the service order-manager.account-manager and the order-manager.account-manager-1 version. These names are derived automatically from the service name and deployment number provided by OpenShift. Let’s select an Aggregation Interval of 10 seconds.

Hawkular APM Services Tab with One Service

Introducing the Blue Deployment and the Canary Release

Now, we’ll create a new build and new application for our "Blue" deployment. We first set the current deployment to have 4 replicas, so that we’ll have 80% of the traffic going to the old version, and 20% going to the new one.

cd account-manager/
oc scale deploymentconfig account-manager --replicas=4
oc new-build --binary --name=account-manager-blue -l app=account-manager-blue
oc start-build account-manager-blue --from-dir=. --follow
oc new-app \
  account-manager-blue \
  -l app=account-manager-blue \
  HAWKULAR_APM_URI='http://hawkular-apm' \
  HAWKULAR_APM_USERNAME='admin' \
  HAWKULAR_APM_PASSWORD='password'

This is the expected output for the previous command:

--> Found image 3f61c6a (37 seconds old) in image stream account-manager-blue under tag "latest" for "account-manager-blue"

    * This image will be deployed in deployment config "account-manager-blue"
    * The image does not expose any ports - if you want to load balance or send traffic to this component
      you will need to create a service with 'expose dc/account-manager-blue --port=[port]' later

--> Creating resources with label app=account-manager-blue ...
    deploymentconfig "account-manager-blue" created
--> Success
    Run 'oc status' to view your app.

We should also have a new service displayed on OpenShift console. Note the new account-manager-blue deployment, with 1 pod.

OpenShift with Blue/Green deployments for Account Manager

Our Account Manager microservice is a Vert.x Verticle, and reacts on messages sent to AccountManager.getAccount. As we are using a clustered event bus, our new account-manager-blue service will join the cluster and start replying to those messages. The Vert.x clustered bus will take care of load balancing the processing of the messages for us.

Because of that, we should already have some data being captured and reported to our Hawkular APM server. On the Services screen, add the new service order-manager.account-manager-blue, version order-manager.account-manager-blue-1 to the graphic. It should then look similar to this:

Hawkular APM Services Tab with Blue and Green services

We should see the bar for our blue deployment (orange bar on the image) to be around a quarter the size of the old version (blue bar on the image). As the number of errors is in line with what we expect, we can now reduce the traffic to the old version by adding more replicas to the new version and decreasing replicas from the old one:

oc scale deploymentconfig account-manager-blue --replicas=3
oc scale deploymentconfig account-manager --replicas=2

Let’s leave it running for a minute or so, and we should see that the new version is now responsible for processing around 60% of the traffic, while the old version is now processing around 40%. If it still looks good, we’ll make it 80%/20%:

oc scale deploymentconfig account-manager-blue --replicas=4
oc scale deploymentconfig account-manager --replicas=1

Eventually, we would want to decommission the old version:

oc scale deploymentconfig account-manager-blue --replicas=5
oc scale deploymentconfig account-manager --replicas=0

Let’s now simulate a new build for our Account Manager microservice, to be deployed on the green service (account-manager):

cd account-manager/
oc start-build account-manager --from-dir=. --follow

Once it’s finished, we can direct some of the traffic to this new version:

oc scale deploymentconfig account-manager --replicas=1
oc scale deploymentconfig account-manager-blue --replicas=4

After a few seconds, we should see the new version as a valid option on our Services screen:

New Green version

Let’s add it to the graphic, and watch the errors. If this version looks sane, we can direct more traffic to it:

oc scale deploymentconfig account-manager --replicas=3
oc scale deploymentconfig account-manager-blue --replicas=2

And eventually, direct all traffic to it:

oc scale deploymentconfig account-manager --replicas=5
oc scale deploymentconfig account-manager-blue --replicas=0

On the next screenshot, we can infer the following from the graphic:

  • We started with 100% traffic on order-manager.account-manager-1 (green deployment, blue bar)

  • We directed some traffic to order-manager.account-manager-blue-1 (blue deployment, orange bar)

  • We switched off order-manager.account-manager-1 (green deployment, blue bar)

  • We deployed the new version order-manager.account-manager-2 (green deployment, green bar)

  • We directed some traffic to order-manager.account-manager-2 (green deployment, green bar)

  • We switched off order-manager.account-manager-blue-1 (blue deployment, orange bar)

Final Result

Closing notes

On a real world scenario, the manual steps we did would be automated, potentially making your Continuous Deployment scripts read data from Hawkular APM in order to decide whether to scale up or down a canary release. Thanks to Vert.x, we were able to completely abstract the load balancer, controlling the percentage of traffic by adjusting the number of replicas on the Green/Blue deployments, but it’s also possible to achieve similar effects with other stacks. The OpenShift Hello World Microservices Architecture shows another way this can be achieved.

Thanks to OpenShift, we were able to instantly see the performance of new versions, comparing it with previous versions. Outside of OpenShift, we would be able to get the same effect by properly setting the environment variables HAWKULAR_APM_SERVICE_NAME and HAWKULAR_APM_BUILDSTAMP.

Try it out and let us know about your experiments! Should you face any issues, or if you want to talk to us, join us at #hawkular on irc.freenode.net.





Hawkular Metrics 0.24.0 - Release

08 February 2017, by Stefan Negrea

I am happy to announce release 0.24.0 of Hawkular Metrics. This release is anchored by a new tag query language and general stability improvements.

Here is a list of major changes:

a1 = 'bcd' OR a2 != 'efg'
a1='bcd' OR a2!='efg'
a1 = efg AND ( a2 = 'hijk' OR a2 = 'xyz' )
a1 = 'efg' AND ( a2 IN ['hijk', 'xyz'] )
a1 = 'efg' AND a2 NOT IN ['hijk']
a1 = 'd' OR ( a1 != 'ab' AND ( c1 = '*' ) )
a1 OR a2
NOT a1 AND a2
a1 = 'a' AND NOT b2
a1 = a AND NOT b2

Hawkular Alerting - Included

Hawkular Metrics Clients

Release Links

A big "Thank you" goes to John Sanda, Matt Wringe, Michael Burman, Joel Takvorian, Jay Shaughnessy, Lucas Ponce, and Heiko Rupp for their project contributions.





Hawkular Metrics - 2017 Roadmap

07 February 2017, by Stefan Negrea

About

Just like last year, Hawkular Metrics contributors have been working for the past few weeks on the roadmap for the upcoming year. The goal is to give clarity on the project direction, serve as a planning tool for releases, and show our strong commitment to open source.

For those not familiar with Hawkular Metrics, the project is a high performance and high availability storage engine for large volume metric data. Cassandra is the storage engine because of its flexible data model well suited for time-series data storage and linear scalability with no single point of failure. The distribution includes Hawkular Alerting, an alerts module responsible for defining conditions rules over data events and fire alerts that can be sent by several action plugins.

Review - 2016

First, let's review the progress against the roadmap set at the beginning of last year. Due to the competitive space for metrics, metrics storage, and alerting, the community had to constantly balance the plan execution while reacting to community requests. While the majority of goals set to accomplish were completed, there are some carried over to this year and some indefinitely postponed.

The project was updated to run on Cassandra 3.x in May (release 0.15.0). The native Grafana integration was released in July (release 0.17.0) and it is now an independent project. And lastly, we improved the developer support by creating distributions that include Metrics, Alerting and Cassandra pre-deployed inside Wildlfly to make it extremely simple to download and get the project running.

The pre-computed aggregates were postponed to this year due to the major work done for compressing data at rest released in October (release 0.20.0). The histogram metrics were also carried-over to this year.

Roadmap - 2017

If you have any other suggestion or would like to contribute to the project, please contact us; feedback is more than welcomed.





Older posts:

RSS Feed

redhatlogo-white

© 2016 | Hawkular is released under Apache License v2.0