Hawkular - Alerts and Notifications for Elasticsearch using Hawkular Alerting

Elasticsearch Alerter

Hawkular Alerting includes a new Alerter capable to query Elasticsearch servers and represent documents as Hawkular Events.

Managing Elasticsearch documents as Events allows Hawkular Alerting to define rich and complex conditions using EventConditions.

Triggers tagged with Elasticsearch will be used to schedule a periodic query to an Elasticsearch server. Trigger’s context will be used to define the interval between queries, additional query filtering or mapping options that will be used by the alerter to convert documents into events.

For example,

    "tags": {
      "Elasticsearch": "Localhost instance" (1)
    },
    "context": {
      "timestamp": "@timestamp", (2)
      "interval": "30s", (3)
      "index": "log",  (4)
      "filter": "{\"match\":{\"app\":\"AppA\"}}", (5)
      "mapping": "level:category,@timestamp:ctime,message:text,app:dataId,index:tags" (6)
    }

Elasticsearch tag is used by the alerter to schedule a query from this trigger
Elasticsearch field "@timestamp" will be used to fetch documents in regular intervals
Interval between queries will be 30 seconds
Documents are fetched from log index
Optional filtering to refine the query
Mapping between Elasticsearch document fields and Hawkular Events

The full reference of the Elasticsearch Alerter can be found under Elasticsearch Alerter.

Elasticsearch Plugin

Hawkular Alerting also includes a new Action Plugin capable to transform Hawkular Events and Alerts into Elasticsearch documents.

The Elasticsearch Plugin properties allows to define how an Event or Alert will be transformed in a document and under which index and type will be stored.

For example,

    {
      "actionPlugin": "elasticsearch",
      "actionId": "write-partial-alert",
      "properties": {
        "index": "alerts_summary",  (1)
        "timestamp_pattern": "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ",   (2)
        "transform": "{\"tenantId\":\"tenant\",\"ctime\":\"timestamp\",\"text\":\"trigger\",\"context\":{\"interval\":\"fetch-interval\"},\"evalSets\":\"details\"}" (3)
      }
    }

Index used to store documents transformed by this plugin
This timestamp_pattern indicates that all timestamp fields of Events/Alerts should be transformed into strings using this pattern
JSON → JSON mapping to convert Events/Alerts into document using JOLT Shiftr

The full reference of the Elasticsearch Plugin can be found under Elasticsearch Plugin.

Logging Example

An small but rich example can be found under Elasticsearch example.

This example simulates different logs entries that are stored in Elasticsearch.

These entries define a log level (INFO, WARN or ERROR), a source application (AppA, AppB and AppC) and in the body message describes some reference about the subcomponent related (Security, Frontend or Backend).

These logs are stored in Elasticsearch with the following format:

{
    "@timestamp":"2017-04-05T14:23:13.360447+0200",
    "level":"ERROR",
    "app":"AppA",
    "message":"Message 6167 from Security"
}

In the example a first trigger is defined to fetch documents just for AppA, for this, the in the context we define a match filter as

    "context": {
      "timestamp": "@timestamp",
      "filter": "{\"match\":{\"app\":\"AppA\"}}", (1)
      "interval": "30s",
      "index": "log",
      "mapping": "level:category,@timestamp:ctime,message:text,app:dataId,index:tags" (2)
    }

From all possible documents, only AppA are interested for this trigger
Note that app field is used as dataId for Events

Once events are mapped we can define an EventsCondition to detect any ERROR log related to Backend components.

      "conditions":[
        {
          "type": "EVENT",
          "dataId": "AppA",
          "expression": "category == 'ERROR',text contains 'Backend'"
        }
      ]

In a similar way, a second trigger is defined in the example to fetch documents for AppB.

    "context": {
      ...
      "filter": "{\"match\":{\"app\":\"AppB\"}}", (1)
    }

From all possible documents, only AppB are interested for this trigger

On this second trigger, we want to detect when we have a suspected high number of WARN messages in the log. There are several ways to map this, in the example we are going to use a Dampening

      "dampenings": [
        {
          "triggerMode": "FIRING",
          "type":"RELAXED_COUNT",
          "evalTrueSetting": 3,
          "evalTotalSetting": 10
        }
      ],
      "conditions":[
        {
          "type": "EVENT",
          "dataId": "AppB",
          "expression": "category == 'WARN'"
        }
      ]

So, this second trigger will alert when more than 3 WARN messages are detected in a rate of 10 messages processed.

Finally, the example defines several ways of notifications: sending an email to administrators and writting back into Elasticsearch the alerts fired.

Alerts and Notifications for Elasticsearch using Hawkular Alerting

Elasticsearch Alerter

Elasticsearch Plugin

Logging Example

Elasticsearch Integration